Developer guide - digitalmethodsinitiative/4cat GitHub Wiki

Interested in contributing to 4CAT or extending it with your own modules? The following pages offer the information you need to do this:

  • Architecture: A high-level overview of 4CAT's software architecture.
  • Extensions: Adding data sources and processors can be done directly in your local 4CAT (and can be submitted via PR to be included here). They can also be made available as extensions that can be installed on any 4CAT instance.
  • How to make a data source: On adding support for data capture from new platforms.
  • How to make a processor: On adding new analytical processors to 4CAT.
  • Input fields: On the API to use for connecting data sources and processors to the web interface.
  • Documentation: Automatically generated documentation from 4CAT's code

4CAT tips

  • Using git to track changes, create a new branch or fork is the best way to developer 4CAT. The Docker 4CAT containers inherit the git branch you cloned and so you can use all of git's functionality inside the containers.
  • Logs are stored by default in the logs folder (/usr/src/app/logs inside Docker). In Docker, this folder is a shared volume so it is accessible/identical via both 4cat_backend and 4cat_frontend (as are data and config).
  • Commands to control 4CAT's backend:
    • python3 4cat-daemon.py status
    • python3 4cat-daemon.py stop
    • python3 4cat-daemon.py start

Developing in Docker

Docker's containerized approach allows you to easily change 4CAT and its environment and redeploy your local application. You can make changes locally in your cloned 4CAT repository, shutdown your running 4CAT instance with docker compose down, and then recreate the application with docker compose -f docker-compose_build.yml up --build -d. This will rebuild your application with your local files following the docker-compose_build.yml file (ensure you use that file as opposed to docker-compose.yml since this file pulls our pre-built images directly from Docker Hub). Note: if you wish to remove all collected data as well, use docker-compose down -v which removes the volumes as well.

Often it is not necessary to rebuild the entire application. If you only change certain files or create new processors/datasources, you can copy those files directly into your Docker 4CAT containers. docker cp path/to/file 4cat_backend:/usr/src/app/path/to/file will copy a file to the 4cat_backend container. Remember to also copy the same files to the 4cat_frontend container which works as a mirror to the backend. The frontend will generally pick up all changed files automatically so long as you refresh your browser, however, the backend will need to be restarted. You could restart the container in Docker desktop or run docker exec 4cat_backend python3 4cat-daemon.py restart.

You can also connect directly to the containers with either the Docker GUI or the command docker exec -it 4cat_backend /bin/bash where you can run commands, test your python3 environment and files, and even edit files directly. (You could install your preferred text editor if you wish with a command such as apt-get install nano directly in the container.)

Docker tips

  • 4CAT Docker containers do not need to be running to copy files to and from them (helpful if an error causes a container to crash!)
  • docker compose essentially relies on the docker-compose.yml (or docker-compose_build.yml) file and your .env file, but also uses the docker\Dockerfile to build the 4cat image environments and the docker/docker-entrypoint.sh file which runs every time the 4cat_backend container starts.
  • You can open additional ports by adding them to docker-compose_build.yml. For example, if you wished to directly connect to the database from your local machine, just add the line ports: - 5432:5432 to the db container under services and recreate 4CAT allowing you to connect to the database from your local machine.

Helpful Docker commands

Most powerful command: docker exec -it container_name bash

In 4CAT context, docker exec -it 4cat_backend bash and docker exec -it 4cat_frontend bash allow you to connect to either container similar to an ssh tunnel to a server. You can then run any command normally (perhaps requiring apt or apt-get to install additional software).

You can also, from your host machine, copy files into container with docker cp path/to/file container_name:/full/path/to/container/directory/ (e.g., docker cp processors/metrics/my_new_processor 4cat_backend:/usr/src/app/processors/metrics/ to copy your new processor).

Note: updates to code generally need to be copied to both 4cat_backend and 4cat_frontend containers and require restarting to take effect (via the frontend GUI's Command Panel, restarting the containers, or perhaps just the backend e.g., docker exec 4cat_backend python 4cat-daemon.py restart).

Common commands

  1. View container logs docker container logs container_name
  2. Stop running container docker stop container_name
  3. Start stopped container docker start container_name
  4. Remove container docker container rm container_name Useful to remove then recreate with new parameters (e.g. port mappings)
  5. Remove image docker image rm image_name:image_tag Useful if you need to change Dockerfile or docker-entrypoint.sh and rebuild
    • Note: must also remove any containers dependent on image; you could alternately create a new image with a different name:tag