Install and run 4CAT

🎥 A video walkthrough installing 4CAT via Docker can be found on YouTube here.

You can install 4CAT on your local machine or a server (see also our guidelines for what type of hardware you need). This can be useful if you want to capture data from various online platforms, import and process data collected with Zeeschuimer, or otherwise analyse data you've captured previously. This page describes how you can install 4CAT and run it. Here you can review a full list of available data sources available.

Please note that various data sources require additional configuration after installation. It is recommended to go to the Control Panel's "Settings" page after installation to complete 4CAT's configuration.

Install 4CAT via Docker

The recommended method is to use Docker. These simple instructions use Docker to manage your images as well as storage. For server setups, be sure to also read this section. If you wish to aid in development of 4CAT with a Docker setup, read this section.

1. Install Docker Desktop, and start it. Note that on Windows, you may need to ensure that WSL (Windows Subsystem for Linux) integration is enabled in Docker. You can find this in the Docker setting in Settings -> Resources-> WSL Integration -> Enable integration with required distros.

Simple: Use 4CAT's Docker Hub Image

2. Download our docker-compose.yml file located here and our .env file located here (you can use your browser's "Save page as" to download these files*). Make a folder in a stable location to save them in.
   • Optionally edit the .env file to change database passwords, server name, or ports
     • DOCKER_TAG can be set to stable for the latest official release, latest for the current under development version (which may have bugs AND bug fixes), or any previous release.
3. Open a terminal window and navigate to the folder you saved the files in (e.g. cd ~/Documents/4cat to change directories to your Documents/4cat folder).
   • Sometimes downloading the .env file will be renamed env.txt which will not work. ls -a will show you the files in the folder you are currently in. If the file is named env.txt, you can rename it yourself with the command mv env.txt .env (newer Mac OS will not let you name a file starting with a . so you will need to use Terminal and mv env.txt .env from the same directory/folder).
4. Run the command docker-compose up --detach in your Terminal/Command Prompt from the folder where you downloaded the above files
   • You may need to run this command as an administrator by either adding sudo to the beginning of the command (Mac/Linux, i.e. sudo docker-compose up --detach; you will be asked for your password) or right clicking your Command Prompt and using "Run as administrator" (Windows) to open it.
5. Congrats! Open http://localhost/ in your browser of choice to start using 4CAT. You can close the terminal now.

Common issues with Docker

• http://localhost/ opens an incorrect page such as the Docker welcome page (from their starter project). Try pressing F5 to reload or clearing your browser cache and then reloading the page.

Install 4CAT via Docker onto a server

Storage

Depending on your use-case, you may wish to modify how Docker handles your data storage. By default, docker-compose.yml uses Docker volumes. This is easy to deploy, but some find it difficult to backup and restore. You may also wish to store 4CAT files on a larger drive than what you have set up for Docker. You can edit the docker-compose.yml file to use mounts instead of volumes. The docker-compose_build.yml does this by default as an example (it is designed for development and also builds new images based on local 4CAT repository files instead of the images we build and post on Docker Hub). To do this you would change the "volumes" sections of the three containers in the docker-compose.yml file. For example, the following would mount folders in your 4cat/data/ directory to the containers for storage of 4CAT data:

db:
    volumes:
      - ./data/postgres/:/var/lib/postgresql/data/
backend:
    volumes:
      - ./data/datasets/:/usr/src/app/data/
      - ./data/config/:/usr/src/app/config/
      - ./data/logs/:/usr/src/app/logs/
frontend:
    volumes:
      - ./data/datasets/:/usr/src/app/data/
      - ./data/config/:/usr/src/app/config/
      - ./data/logs/:/usr/src/app/logs/

Hosting

With Docker, 4CAT is set to host itself on localhost:80 by default. This can be modified in the .env file located in the main directory. We recommend using Nginx or Apache for both performance and security if you plan on serving it to multiple users and/or at a public domain. SERVER_NAME in the .env file is only used by 4CAT when first installing, afterwards, you can update it via 4CAT's UI to your domain or IP address. Once you set this up, you can further configure 4CAT to take various proxy headers into account via Control Panel -> Settings -> Flask settings -> Use proxy headers for URL.

# Modify SERVER_NAME and/or PUBLIC_PORT to make 4CAT available externally

SERVER_NAME = 4cat.example.com # You could also use your server's IP address
PUBLIC_PORT = 80

# This example would allow you to navigate to http://4cat.example.com (or http://4cat.example.com:80) and access your version of 4CAT

Development and Customization

You can use a local clone of 4CAT in order to develop new datasources and processors. There are additional sections in our wiki on developing as well as hints on doing so with Docker.

Note: you can also of course edit files directly in the Docker containers from the simple install, however, these will be lost if you rebuild the containers.

1. Install Docker.
2. Clone the 4CAT repository, or download the most recent release and unzip it.
3. In a terminal/command prompt, navigate to the folder in which you just installed 4CAT (the folder that contains the docker-compose_build.yml file)
4. Run the command docker-compose -f docker-compose_build.yml build to build from your local 4CAT files
5. If this is the first time you're starting the Docker container, it will take a while for all components to be built.
6. Run the command docker-compose -f docker-compose_build.yml up -d to start your container for the first time.
7. Once this is done, you can access the 4CAT interface via http://localhost:80.
8. You can now edit files locally and rebuild your 4CAT version via docker-compose down then docker-compose up --build -d (or just edit directly in the docker containers, restarting the backend via python3 4cat-daemon.py restart).

Note: if your computer/server is already using some of the same ports that Docker wishes to use, you can modify the .env file in the home directory and change the ports that Docker uses. Any modifications to configuration files will require you to rebuild the docker images with docker-compose -f docker-compose_build.yml up --build.

Install 4CAT manually

If you cannot or don't want to use Docker, you can run 4CAT directly from the code rather than via Docker. This requires more set-up and the manual installation of various dependencies, but can be useful if you want to develop data sources or processors for 4CAT.

Requirements

It is recommended that you run 4CAT on a UNIX-like system (e.g. Linux or MacOS). It will also run under Windows, but the instructions below are written with a UNIX-like in mind. 4CAT further requires Python 3.11 and PostgreSQL 9.5 or higher. Lower versions of either may work, but are not officially supported.

Installation

Clone the repository somewhere:

git clone https://www.github.com/digitalmethodsinitiative/4cat.git

After cloning the repository, copy config/config.ini-example to config/config.ini and edit the file to match your machine's configuration. The various options are explained in the file itself:

cd 4cat/config
cp config.ini-example config.ini
nano config.ini

Next, install the dependencies. On Linux systems that use apt, the following should suffice:

apt install python3-pip libpq-dev python3-dev postgresql postgresql-server-dev-all unzip postgresql-client ffmpeg

Adapt these to your own package manager (e.g. yum, brew, choco) as necessary. From here on, we are working within Python, so it is recommended you create a virtual environment to install Python packages and run 4CAT in. There are several ways to set up a virtual environment, the link earlier in this paragraph lists the best practices.

Within your virtual environment, while in the 4CAT root folder, install the required Python packages:

pip3 install -r requirements.txt

Some of the dependencies may have their own dependencies. For instance, on Windows the pyahocorasick library needs to Microsoft Visual C++ Build Tools to be installed. If you encounter similar issues, please file an issue!

Next, you should make sure a database is available for 4CAT. 4CAT requires a PostgreSQL database to store dataset metadata, the job queue and other assorted data. You should create the database yourself, and add the database login details to config.ini. After doing so, run the following command to create the tables, indices, et cetera, required by 4CAT:

psql --user=[username] --dbname=[database name] < backend/database.sql

Replace [username] and [database name] with the relevant values. You may be prompted for a password. On Windows, the following command should work: psql -U [username] -d [database name] -a -f backend/database.sql

Finally, to make sure everything is in working order, run the following command and follow the instructions:

python3 helper-scripts/migrate.py

You can now run 4CAT!

Running the backend

The backend is run as a daemon that can be started and stopped using the included 4cat-daemon.py script:

python3 4cat-daemon.py start

Other valid arguments are stop, restart and status. Note that if you change any configuration options, you will need to restart the daemon for the changes to take effect. For development/testing it may be helpful to run 4cat-daemon.py interactively with the -i switch (i.e., python3 4cat-daemon.py -i start). This will log output to the terminal as well.

Note: The 4CAT daemon was made to run on a UNIX-like system and the above will not work on Windows. On Windows, the 4CAT daemon will always run interactively, and can be quit by entering 'q' and pressing Enter.

4CAT logs to 4cat.log in the root folder by default.

Running the web tool

The web tool is a Flask app. It is recommended that you run the web tool as a WSGI module: see the Flask documentation for more details. For testing and development, you can run the Flask app locally from the command line. For Mac:

FLASK_APP=webtool flask run

For Windows:

set FLASK_APP=webtool
flask run

With the default configuration, you can now navigate to http://localhost:5000 where you'll find the web tool that allows you to query the database and create datasets. On first visit you will be asked to create an admin account through which the tool can be managed

You may wish to modify some 4CAT settings

Most settings are now accessible in the 4CAT database in the settings table. You can update settings via the 4CAT web interface if you are an admin user by navigating to the "Control Panel" tab at the top of the interface and then via the buttons at the top of the page.

If you are unable to navigate to the 4CAT web interface with default settings (e.g., if you are deploying 4CAT on a server instead of localhost), you can also modify settings directly in the database.

1. Connect to psql
   • psql --user=[username] --dbname=[database name] (changing username and database name as appropriate)
2. View settings
   • SELECT * FROM settings;
   • or perhaps SELECT * FROM settings WHERE name LIKE '%flask%'; to view flask settings
3. Update as needed
   • UPDATE settings SET value = '"my.server.com"' WHERE name = 'flask.server_name';
   • UPDATE settings SET value = '["localhost", "my.server.com"]' WHERE name = 'flask.autologin.hostnames';

Installing 4CAT using previous data

Assuming you have appropriately backed up your prior 4CAT database and dataset files, you can install 4CAT using your prior setup. You will also need to have some configuration files in order to identify what was your previous version of 4CAT and, if desired, for 4CAT to attempt to upgrade to a newer version.

See additional instructions here.