Deploy D‐Genies in webserver mode with virtualenv, gunicorn, nginx and systemd - genotoul-bioinfo/dgenies GitHub Wiki

For this recipe, we will use CentOS 7 as Linux distribution.

Note: All this recipe commands will be done as root user, or prefixed with sudo.

Install D-Genies in a virtualenv

Requirements

We will install D-Genies as root user, but D-Genies will be run by the user dgenies. We create then the user dgenies:

# We add the user
useradd -m dgenies

# We can also set the user password
passwd dgenies

By default, CentOS is installed with SELinux activated and in Enforcing state. We disable it in order to avoid interference with D-Genies installation and configuration.

# We deactivate SELinux (need to reboot to be taken in account)
sed -i 's/^SELINUX=.*/SELINUX=disabled/' /etc/selinux/config 
# Set SELinux in permissive state now! This way, we don't need to reboot.
setenforce 0

We also deactivate the firewall for the same reason

systemctl stop firewalld
systemctl disable firewalld

D-Genies is written for Python 3.5+, which can be installed with the following command:

yum install -y python3 python3-devel python3-pip

D-Genies also needs the following system dependencies.

yum install -y  "@Development tools" time zlib-devel libjpeg-turbo-devel libsq3-devel sqlite-devel

Finally, a good practice it use a virtual environment to install python packages that are not managed by the distribution repository.

yum install -y python-virtualenv

Install D-Genies

First, we create the virtual environment for D-Genies:

virtualenv -p python3 /usr/local/venv/dgenies

Then, we activate it:

source /usr/local/venv/dgenies/bin/activate
# we check which python3 we use
which python3
# must return /usr/local/venv/dgenies/bin/python3

Finally, we install D-Genies.

cd /tmp
git clone https://github.com/genotoul-bioinfo/dgenies.git dgenies.git
cd gdenies.git
python3 setup.py install

You can also install D-Genies from pip, but pip behaviour has changed over time. Recent pip versions do not copy some D-Genies files in the right place anymore. If you want to try, you can use the following command:

python3 -m pip install dgenies

Configure D-Genies

From this step, you can configure D-Genies by editing the /etc/dgenies/application.properties file and run it directly with the dgenies run -m webserver command, but it is not the way to serve wsgi application in production mode.

We also recommend to use a MySQL/MariaDB database instead of the SQLite one use by default by D-Genies

TO DEVELOP

Use gunicorn to serve flask application

We use gunicorn in order to serve D-Genies as wsgi application. We install it in the D-Genies virtualenv:

Install gunicorn

source /usr/local/venv/dgenies/bin/activate
python3 -m pip install "gunicorn>=20.1"

D-Genies can be now launched this way:

sudo -u dgenies /usr/local/venv/dgenies/bin/gunicorn -b 127.0.0.1:8000 'dgenies:launch(debug=True)'

where -b 127.0.0.1:8000 only allows localhost to connect to D-Genies on the port 8000 (with http protocol). Please set your own ip instead if you connect from distant computer, or use 0.0.0.0 to allow everybody.

By connecting on 127.0.0.1:8000 (or the server ip), you will be able to access to D-Genies.

Configure gunicorn

There is many ways to configure gunicorn.

We choose to use a configuration file for gunicorn. We create /etc/dgenies/gunicorn.conf.py file with the following content:

import multiprocessing                                                                                                 

proc_name = "dgenies"
bind = "127.0.0.1:8000"
workers = multiprocessing.cpu_count() + 1
accesslog = "/home/dgenies/logs/gunicorn.access.log"
errorlog = "/home/dgenies/logs/gunicorn.error.log"
timeout = 120 # For huge load

Now it is possible to run D-Genies this way.

sudo -u dgenies /usr/local/venv/dgenies/bin/gunicorn --config=/etc/dgenies/gunicorn.conf.py 'dgenies:launch(debug=True)'

Nginx proxy for gunicorn

Nginx is available in the epel repository. We install it with the following commands:

# We activate epel repository
yum install -y epel-release

# We install nginx
yum install -y nginx

We then configure nginx by following gunicorn guidelines

We create the /etc/nginx/conf.d/dgenies.conf file:

upstream app_server {
    # fail_timeout=0 means we always retry an upstream even if it failed
    # to return a good HTTP response

    # for UNIX domain socket setups
    # server unix:/tmp/gunicorn.sock fail_timeout=0;

    # for a TCP configuration
    server 127.0.0.1:8000 fail_timeout=0;
}

server {
    # if no Host match, close the connection to prevent host spoofing
    listen 80 default_server;
    return 444;
}

server {
    # use 'listen 80 deferred;' for Linux
    # use 'listen 80 accept_filter=httpready;' for FreeBSD
    listen       80 deferred;
    listen       [::]:80 deferred;
    client_max_body_size 4G;

    # set the correct host(s) for your site
    server_name  localhost; # Adjust it

    keepalive_timeout 5;

    # path for static files
    root /var/www/dgenies;

    location / {
        # checks for static file, if not found proxy to app
        try_files $uri @proxy_to_app;
    }

    location ~* /gallery/*\.(png|svg) {
        # checks for images file for gallery, if not found proxy to app
        # Adjust it with D-Genies config path; also set permissions (read + traversal for nginx user)
        alias /home/dgenies/.dgenies/data/gallery/; 
        try_files $uri @proxy_to_app;
    }

    location @proxy_to_app {
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header Host $http_host;
        # we don't want nginx trying to do something clever with
        # redirects, we set the Host: header above already.
        proxy_redirect off;
        proxy_pass http://app_server;
    }
}

We also configure D-Genies static data.

cd /var/www/dgenies

# We remove the apache wgsi file installed by D-Genies
rm dgenies.wsgi

# We create a link to static parts of D-Genies in order to make nginx serve them
ln -s /usr/local/venv/dgenies/lib/python3.6/site-packages/dgenies-1.3.0-py3.6.egg/dgenies/static .

When everything is configured, we restart nginx

systemctl restart nginx

D-Genies as a systemd service

gunicorn documentation

We create a systemd service file /etc/systemd/system/dgenies.service with following contents:

[Unit]
Description=D-Genies gunicorn daemon
# Requires=dgenies.socket
After=network.target remote-fs.target nss-lookup.target

[Service]
Type=notify
User=dgenies
Group=dgenies
# another option for an even more restricted service is
# DynamicUser=yes
# see http://0pointer.net/blog/dynamic-users-with-systemd.html
RuntimeDirectory=dgenies
ExecStart=/usr/local/venv/dgenies/bin/gunicorn --config=/etc/dgenies/gunicorn.conf.py 'dgenies:launch()'
ExecReload=/bin/kill -s HUP $MAINPID
#ExecStop = /bin/kill -s TERM $MAINPID
KillMode=mixed
TimeoutStopSec=5
PrivateTmp=true

[Install]
WantedBy=multi-user.target

We reload then the systemctl daemon in order to take in account our service:

systemctl daemon-reload

Now we can start D-Genies:

systemctl start dgenies
systemctl enable dgenies

D-Genies can be stopped this way:

# We stop the flask application
systemctl stop dgenies

# We can also stop the local scheduler launched by the flask app.
sudo -u dgenies /usr/local/venv/dgenies/bin/dgenies clear -c

TODO:

  • systemd service for local scheduler
  • systemd timer for cleaning