Web Development with Python, Django, uWSGI and PostgreSQL - giobim/Some-help-from-my-Friends GitHub Wiki

In Part 1, we will prepare our linux (native or WSL) runtime for Simple, but powerful way, to develop 3-Tiers Web Applications.

Prerequesites

A linux system (Ubuntu 18.04 LTS) native or using Windows Subsystem Linux feature Windows: Installing WSL feature.

Set up the environment for development with python

Install the virtualenv, I show a procedure here under Ubuntu using virtualenvwrapper and set up the local environment for the user:

$ sudo apt install python3 python-pip3 build-essential cmake

$ sudo pip3 install virtualenvwrapper

Somewhere in your ~/.bashrc, include:

export VIRTUALENVWRAPPER_PYTHON=/usr/bin/python3
export WORKON_HOME=$HOME/.virtualenvs
export VIRTUALENVWRAPPER_VIRTUALENV_ARGS=' -p /usr/bin/python3 '
source /usr/local/bin/virtualenvwrapper.sh

Activate ypur environment:

$ source $HOME/.bashrc

Set up Django and generate your project

Set up an python virtual environment for django developemnt:

$ mkvirtualenv django

(django) $ pip3 install django uwsgi

(django) $ django-admin startproject mysite

  Notice: Couldn't set permission bits on /mnt/d/Documents/Workspaces/Experiments/django_test/mysite/manage.py. You're probably using an uncommon filesystem setup. No problem.
  Notice: Couldn't set permission bits on /mnt/d/Documents/Workspaces/Experiments/django_test/mysite/mysite/__init__.py. You're probably using an uncommon filesystem setup. No problem.
  Notice: Couldn't set permission bits on /mnt/d/Documents/Workspaces/Experiments/django_test/mysite/mysite/settings.py. You're probably using an uncommon filesystem setup. No problem.
  Notice: Couldn't set permission bits on /mnt/d/Documents/Workspaces/Experiments/django_test/mysite/mysite/urls.py. You're probably using an uncommon filesystem setup. No problem.
  Notice: Couldn't set permission bits on /mnt/d/Documents/Workspaces/Experiments/django_test/mysite/mysite/wsgi.py. You're probably using an uncommon filesystem setup. No problem.

(django) $ cd mysite

(django) $ python manage.py migrate

  Operations to perform:
    Apply all migrations: admin, auth, contenttypes, sessions
  Running migrations:
    Applying contenttypes.0001_initial... OK
    Applying auth.0001_initial... OK
    Applying admin.0001_initial... OK
    Applying admin.0002_logentry_remove_auto_add... OK
    Applying admin.0003_logentry_add_action_flag_choices... OK
    Applying contenttypes.0002_remove_content_type_name... OK
    Applying auth.0002_alter_permission_name_max_length... OK
    Applying auth.0003_alter_user_email_max_length... OK
    Applying auth.0004_alter_user_username_opts... OK
    Applying auth.0005_alter_user_last_login_null... OK
    Applying auth.0006_require_contenttypes_0002... OK
    Applying auth.0007_alter_validators_add_error_messages... OK
    Applying auth.0008_alter_user_username_max_length... OK
    Applying auth.0009_alter_user_last_name_max_length... OK
    Applying auth.0010_alter_group_name_max_length... OK
    Applying auth.0011_update_proxy_permissions... OK
    Applying sessions.0001_initial... OK

(django) $ python manage.py createsuperuser

(django) $ python manage.py runserver

Test the application by opening your browser at http://localhost:8000/.

Test http://localhost:8000/admin, i.e. log into admin with the username and password you userd for the superuser, when you are finished go back to your shell and hit Ctrl-C.

Set up uWSGI

We use uWSGI from the same django development environment.

Note that uWSGI is a full fledged http server that can and does work well on its own. I've used it in this capacity several times and it works great. If you need super high throughput for static content, then you have the option of sticking nginx in front of your uWSGI server. When you do, they will communicate over a low level protocol known as uwsgi. See the Going into production chapter to configure Nginx in front of uWSGI.

For development we simply use uWSGI to serve static content as well. In Django, we need to collect all the static files from every components by first adding the following line at the end of the mysite/settings.py:

STATIC_ROOT = os.path.join(BASE_DIR, "static")

Then, you can issue the collectstatic which will create directory static under your project path. You can change the location and name of the file in the settings.py and rerun the action anytime.

(django) $ python manage.py collectstatic

119 static files copied to . . .

Note that if you add/install new components or extensions to django framework, you probably will have to collect the static files again.

At last, start uWsgi server application and to load your module and serve static files under the path /static:

(django) $ uwsgi --static-map /static=$PWD/static --http :8000 --module mysite.wsgi

[uwsgi-static] added mapping for /static => /home/giobim/Workspaces/Experiments/django_test/mysite/static
*** Starting uWSGI 2.0.18 (64bit) on [Thu Jan 16 17:30:38 2020] ***
compiled with version: 5.4.0 20160609 on 16 January 2020 15:30:24

. . .

Python version: 3.5.2 (default, Oct  8 2019, 13:06:37)  [GCC 5.4.0 20160609]
*** Python threads support is disabled. You can enable it with --enable-threads ***
Python main interpreter initialized at 0x248e830
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 72904 bytes (71 KB) for 1 cores
*** Operational MODE: single process ***
WSGI app 0 (mountpoint='') ready in 0 seconds on interpreter 0x248e830 pid: 20960 (default app)
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI worker 1 (and the only) (pid: 20960, cores: 1)

Note: Under Windows, the firewall will kick-in if it is activated and shows a dialog, set the flags as you wish.

You can now open http://localhost:8000 in your browser.

If you edit a file then the framework will try to compile and active your code automatically, so always have an eye on the terminal to see for any compilation errors.

Hit Ctrl-C when fishied testing ....

Exit or switch between dev environments

You can switch between python virtual dev environment issuing:

(django) $ workon otherenv

(otherenv) $

Deactivate the presend environment:

(otherenv) $ deactivate

$

Less than 15 minutes, not bad for a start ...

Install PostgreSQL (optional)

$ sudo apt install postgresql postgresql-client postgresql-contrib

Open TCP/IP Connections

Like a good database should, PostgreSQL comes in a very locked-down state security-wise. It is up to you to increase its surface area just enough to operate it effectively, from only the machines that need to access it, and no more. I connect to PostgreSQL from other machines on the network, so let’s make PostgreSQL listen on more than just localhost.

$ sudo nano -w /etc/postgresql/9.5/main/postgresql.conf

Ctrl-W to do a find for “listen_”, and change this line to:

listen_addresses = '*'

Start the server (under WSL):

$ sudo service postgresql start

 * Starting PostgreSQL 9.5 database server                                                             [ OK ]

Note: This will not survive the restart of the Windows machine, you will have to execute sudo service postgresql start every time you start a WSL session. We could create a start_serv.sh that do this automatically if the DB service isn't started already, i.e.

#!/bin/bash

DB_STATUS=$(service postgresql status| cut -f4 -d' ')
if [ $DB_STATUS != 'online' ] ; then
        sudo service postgresql start
else
        echo 'Postgresql already Listening...'
fi
uwsgi --static-map /static=$PWD/static --http :8000 \
--module mysite.wsgi

Change Password of Postgres User

$ sudo -u postgres psql template1

Also following proper security practices, Postgres runs as the less-privileged “postgres” user. The above is a good example of the Switch-User Do [sudo] command being used to perform an operation as a user other than Root. The -u postgres parameter is telling Linux to, for the life of this next command, use the Postgres user rather than Root. So, the entire command above is saying, “Execute the psql template1 command as the user postgres.” psql template1 launches the Postgres command-line app [psql] and logs into the ‘template1’ database. When switching from Root to a user of lesser privilege, you will not be authenticated.

We must do a sudo at this stage rather than simply passing username using the psql -U postgres method because by default Ident Authentication is the active authentication method for local connections. Ident Authentication auto-passes your Linux credentials to Postgres. Think of it as “Windows Authentication” in a SQL Server environment. This isn’t necessarily desirable in a web scenario because most web apps are configured to use local SQL users with an MD5-encrypted password handshake. We are going to change authentication method to MD5 in a sec.

Enter this SQL code to change the password of the postgres user:

ALTER USER postgres WITH ENCRYPTED password 'your_password';

Enter \q to exit.

Host-Based Authentication

Edit Postgres’ Host-Based Authentication file:

$ nano /etc/postgresql/9.5/main/pg_hba.conf

Hit Esc-/ to go to the bottom of the file.

Comment out the first line that start with “local”. Then, replace “host all” line under IPv4 as follows, changing IP address accordingly for your network:

# TYPE  DATABASE    USER        CIDR-ADDRESS          METHOD
# Database administrative login by UNIX sockets
# local   all         postgres                          ident sameuser
# "local" is for Unix domain socket connections only
local   all         all                               md5 
# IPv4 local connections:
host    all         all         192.168.1.0/24        md5
# IPv6 local connections:
host    all         all         ::1/128               md5

I’ve also moved the column header line [TYPE DATABASE USER] above all of the other lines.

What we have done above is similar to what is called “SQL Authentication” in the Microsoft world, IOW, all users stored locally in the SQL database. We’re not using IPv6, so forget that line for now. The line beginning with “local” is saying, “All SQL users have access to every database and must authenticate by supplying a password [which is encrypted with the MD5 hash algorithm].”

The first line beginning with “host” is saying, “All remote users must have an IP address of 192.168.1.x and must authenticate by supplying a password [encrypted with MD5] for a local SQL user.” Restart DB Services

$ sudo service postgresql restart

 * Restarting PostgreSQL 9.5 database server                                                   [ OK ] 
(django) $

Enable your Django app to use PostgreSQL

There are 5 steps involved in adding a Postgres database to your app:

Install requirements in django
Create a new user in Postgres (recommended)
Create a new database and give the new user access
Update your settings.py file to connect to the new database
Migrate & test!

Activate development environment and install requirements

$ workon django

(django) $ sudo apt install libpq-dev

Reading package lists... Done
Building dependency tree

. . .

Processing triggers for libc-bin (2.23-0ubuntu11) ...

(django) $ pip3 install psycopg2

Collecting psycopg2
  Using cached https://files.pythonhosted.org/packages/84/d7/6a93c99b5ba4d4d22daa3928b983cec66df4536ca50b22ce5dcac65e4e71/psycopg2-2.8.4.tar.gz
Building wheels for collected packages: psycopg2
  Building wheel for psycopg2 (setup.py) ... done
  Created wheel for psycopg2: filename=psycopg2-2.8.4-cp35-cp35m-linux_x86_64.whl size=359738 sha256=8ffe0be39bc000cad5ce725aeff38d052258594142926aa7429db3612a997616
  Stored in directory: /home/giobim/.cache/pip/wheels/7e/5b/53/30085c62689dcfce50c8f40759945a49eb856af082e9ebf751
Successfully built psycopg2
Installing collected packages: psycopg2
Successfully installed psycopg2-2.8.4
You should consider upgrading via the 'pip install --upgrade pip' command.
(django) $

This isn’t strictly required, but it is recommended — using a different user for each app will help keep data more secure and limit the amount of damage that can be done to your databases.

To do this, open Postgres in the terminal by typing:

sudo -u postgres psql

Create your user (with the username and password of your choice):

CREATE USER mysiteuser WITH PASSWORD 'mysitepassword';

Make sure you put the password in single quotes — Postgres is expecting a string there.

Create a new database and give the new user access:

Now you need to create the database your app will be connecting to. All we’re doing in this step is creating the data structure to link to — the actual tables will be created by Django when you migrate in step 4.

With Postgres running in your terminal, use the CREATE DATABASE command — to make your life easier, I recommend making your new, dedicated user the owner of the database (more on that in a moment):

CREATE DATABASE mysite WITH OWNER mysiteuser;

In Postgres, the owner of the database has full control over it, to perform any operation on current or future tables.

However, there are a couple of alternatives, depending on your situation:

If for any reason you wanted to grant specific permissions to your user (if you’re storing data in the database that you don’t want to have the app access but you do want the user to access, for example), use GRANT to give the user access to specific tables after migrating in step 4. Beware, however that this can produce errors down the road if the user doesn’t have the right access specifically assigned.
Postgres has the concept of a schema, which in this context means a collection of tables. It also has an automatically created schema called public that includes all tables. So, you could write a command like this to give the user “all access” to the public schema:

GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO mysiteuser;

Update your mysite/settings.py file to connect to the new database

As you probably guessed from the name of this step, we are going to leave the terminal for a moment and go into your project’s settings.py file to give our Django app new instructions about where to look for the database.

When you initiate a project, Django automatically generates a SQLite database, called db.sqlite3 in your project root, and connects it to your app, with code in settings.py that looks something like this.

. . .

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
    }
}

. . .

First, change the engine so that it uses the postgresql_psycopg2 backend instead of the sqlite3 backend. For the NAME, use the name of your database (mysite in our example). We also need to add login credentials. We need the username, password, and host to connect to. We’ll add and leave blank the port option so that the default is selected:

. . .

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': 'mysite',
        'USER': 'mysiteuser',
        'PASSWORD': 'password',
        'HOST': 'localhost',
        'PORT': '',
    }
}

. . .

When you are finished, save and close the file.

Migrate the Database and Test your Project

Now that the Django settings are configured, we can migrate our data structures to our database and test out the server.

We can begin by creating and applying migrations to our database. Since we don’t have any actual data yet, this will simply set up the initial database structure:

(django) cd mysite
(django) python manage.py makemigrations
(django) python manage.py migrate

After creating the database structure, we can create an administrative account by typing:

(django) python manage.py createsuperuser

You will be asked to select a username, provide an email address, and choose and confirm a password for the account.

Once you have an admin account set up, you can test that your database is performing correctly by starting up the Django development server:

(django) uwsgi --static-map /static=$PWD/static --http :8000 --module mysite.wsgi

Configuration

In production we are using Web Server frontend that take care to serve static content, off-load SSL and present information for us. Note that it is also common to use a Load Balancer (LB) like haproxy to offload SSL and balance requests.

We will have to put all the application static files somewhere the web server(s) has (have) access to, i.e. a local (shared) filesystem. Note that if we are planning to scale our service to many clients, the different tiers should be able to communicate with each others efficiently. Thus, we are using caching, distribution and replication throughout the architectiure.

Every tiers is highly available, secured and efficiently cached using ressource pools for data bases, in-memmory cache, distributed object store and shared filesystem of static storage.

But this is part of going to production and a long road ahead of us described in Part 3. Simple and powerful way to develop 3-tiers applications.

That's all Folks!