Container Environment - irtlab/lost GitHub Wiki

This page describes the Docker-based container environment that can instantiate a hierarchy of LoST servers, responders, and publishers. This environment can be used for prototyping, testing, and performance evaluation.

The Model

The entire hierarchy is described by a model stored in model.json. This file represents the single source of truth. All other files (Docker compose file, data to include in Docker containers) are generated from it. The model file consists of a hierarchy (tree) of records. Each record is a JSON object with a few attributes. If a record has children records, as is the case for the root or US servers, for example, then the record has a property called "children", for example:

{
    "$port"      : 5000,
    "server"     : true,
    "$path"      : "root",
    "loadShapes" : "data/world/*.geojson",
    "children"   : {
        "United States": {
            "children": {
                "New York": {
                    "server" : true,
                    "osmId"  : "relation/61320",
                    "$path"  : "ny"
                },
            }
        }
    }
}

Property names starting with $ represent attributes that have automatically generated values. For example, the property $port will be set to a unique port number if not defined in a record. Some properties are inherited from parents if not defined in a record. For example, the property "server" will be implicitly set to true in the "United States" record because the parent (root) server has it set to true.

The following properties are supported:

Name	Inherited	Type	Description
$port	No	number	Automatically generated unique port number the container can bind
server	Yes	boolean	true if this is a LoST server, false otherwise
$path	No	string	Path to the container from the root
loadShapes	Yes	string	A glob pattern of .geojson files to load into the server
children	No	object	A dictionary (object) of children objects
profile	Yes	string	A profile to apply to the generated docker compose service
redirect	Yes	boolean	Set to "true" to enable the redirection mode on a LoST server

Model Utility

There is a new command line program called lost-model for working with the model. Run the program without any parameters to see what it can do:

$ lost-model
Usage: lost-model [OPTIONS] COMMAND [ARGS]...

  LoST model

Options:
  -m, --model TEXT       JSON model file
  -u, --url-prefix TEXT
  --help                 Show this message and exit.

Commands:
  compose   Generate Docker compose file
  fetchall  Fetch all OSM objects from the model

Generating Docker Compose File

The lost-model command "compose" generates a Docker Compose YAML file out of the model and a Jinja2 template file stored in compose.yml.jinja2. The compose file is written to compose.yml by default. This file is stored in Git.

$ lost-model compose
Loading model file /home/janakj/lost/model.json...
Reading template /home/janakj/lost/compose.yml.jinja2...
Writing output to /home/janakj/lost/compose.yml...
Writing URL map to /home/janakj/lost/url-map.json

The command also generates a URL mapping JSON file in url-map.json by default. This file maps geographic objects (polygons) to LoST server URLs. This is included in the LoST server Docker container. The LoST server loads this file into the mapping table when started for the first time.

Fetching OpenStreetMap Polygons

The LoST server Docker container includes GeoJSON boundary polygons for known countries and U.S. states downloaded from OpenSteetMap. Run lost-model fetchall to download all polygons mentioned in the model file:

$ lost-model fetchall
Loading model file /home/janakj/lost/model.json...
Downloading United States.../home/janakj/lost/data/world/us.geojson
Downloading Alabama.../home/janakj/lost/data/world/us/al.geojson
Downloading Alaska.../home/janakj/lost/data/world/us/ak.geojson
Downloading Arizona.../home/janakj/lost/data/world/us/az.geojson
Downloading Arkansas.../home/janakj/lost/data/world/us/ar.geojson
Downloading California.../home/janakj/lost/data/world/us/ca.geojson
...

The polygons will be saved under data/world in the Git repository. The data is included in the LoST server docker contain. The LoST server can be configured to load selected polygons into its database using the environment variable LOAD_SHAPES.

Docker Containers

The Docker Compose file compose.yml creates the following entities:

One LoST resolver
One root LoST server for the entire world
One LoST server for the United States
One LoST server for each EU country

The LoST resolver is configured to send all LoST requests to the root (world) LoST server. The root server delegates to the main U.S. server and the individual LoST servers for other top-level countries. The main U.S. server delegates to the LoST servers for individual U.S. states.

Each entity uses two containers with names following the pattern "lost-db-" and "lost-{resolver,server}-". The former runs the entity's private PostgreSQL database. The latter runs the resolver, server, or other entity type.

The PostgreSQL servers can be accessed from the host running the containers as follows:

$ psql "dbname=lost host=/run/postgresql/server/root user=lost-server"

The UNIX domain sockets of all server-specific PostgreSQL databases are exposed under /run/postgresql/server/*. The resolver's PostgreSQL database is accessible through the UNIX domain socket in /run/postgresql/resolver.

To use the command docker compose up -d (in the folder with compose.yml) to start the containers. By default, only the LoST resolver and the root LoST server containers will be started:

$ docker compose up -d
[+] Running 3/4
 ⠹ Network data_default        Created
 ✔ Container lost-db-resolver  Started
 ✔ Container lost-db-root      Started
 ✔ Container lost-server-root  Started

To start additional servers, e.g., all the U.S. LoST servers, pass the argument --profile us to the command:

$ docker compose --profile us up -d
[+] Running 79/107
...
 ✔ Container lost-db-us-nc      Started
 ✔ Container lost-db-us-va      Started
 ✔ Container lost-db-us-co      Started
 ✔ Container lost-db-us-fl      Started
 ✔ Container lost-db-us-wy      Started
 ✔ Container lost-db-us-hi      Started
 ✔ Container lost-db-us-az      Started
 ✔ Container lost-db-us-ca      Started
 ✔ Container lost-db-us-ut      Started
 ✔ Container lost-db-us-id      Started
 ✔ Container lost-db-us-wa      Started
...

The compose file currently defines two profiles: "us" and "eu". The profile "us" starts the main U.S. LoST server and one additional LoST server per U.S. state. The profile "eu" starts one LoST server per EU country.

Purging Database Data

The PostgreSQL database containers store data in named volumes. The volumes are persistent, i.e., they will remain on the system even when the corresponding database container is removed or recreated. To list all volumes use:

$ docker volume ls
DRIVER    VOLUME NAME
local     lost_db-resolver
local     lost_db-root

If you want to delete all volumes (after deleting the corresponding containers with docker compose down), run docker volume prune -a (the -a flag is necessary because the volumes are named).

Each volume has a "role" label which is set to one of "resolver", "server", "publisher", "responder". The labels can be used, for example, to only delete LoST server volumes:

docker volume prune -a --filter "label=role=server"

The above command will wipe out all LoST server data, but it will retain the databases of LoST resolver and other entity types.

Rebuilding the Containers

Run make docker to rebuild all LoST-related containers. You may need to rebuild if you:

Modify the Python source code
Refresh (re-download) GeoJSON files from OpenStreetMap
Modify the database schema in scripts/sql
Update the url-map.json file by running lost-model compose

Running a Development Instance

Suppose you want to replace the LoST server for us-ny with your own development instance running on the host so that the root US server would forward requests for the NY state to your development instance:

Stop the us-ny LoST server container: docker stop lost-us-ny
Lookup the server's port number in the compose.yml file
Start your development server and configure it to listen the port and connect to the us-ny server's database: lost-server --db-url "user=lost-server host=/run/postgresql/server/us/ny dbname=lost" start -p 5132