Backend - geosolutions-it/digital-twin-toolbox GitHub Wiki

Requirements

  • Docker.
  • Poetry for Python package and environment management.

Local Development

./scripts/compose.sh --workers vector,point-cloud -- up -d

The stack also starts a redis service (used as the Celery broker/result backend) and the selected background workers, which perform the asset conversions.

  • Now you can open your browser and interact with these URLs:

Frontend, built with Docker, with routes handled based on the path: http://localhost

Backend, JSON based web API based on OpenAPI: http://localhost/api/

Automatic interactive documentation with Swagger UI (from the OpenAPI backend): http://localhost/docs

Traefik UI, to see how the routes are being handled by the proxy: http://localhost:8090

Dozzle, container log viewer: http://localhost:9999

Note: The first time you start your stack, it might take a minute for it to be ready. While the backend waits for the database to be ready and configures everything. You can check the logs to monitor it.

To check the logs, run:

docker compose logs

To check the logs of a specific service, add the name of the service, e.g.:

docker compose logs backend

If your Docker is not running in localhost (the URLs above wouldn't work) you would need to use the IP or domain where your Docker is running.

Backend local development, additional details

General workflow

By default, the dependencies are managed with Poetry, go there and install it.

From ./backend/ you can install all the dependencies with:

$ poetry install

Then you can start a shell session with the new environment with:

$ poetry shell

Make sure your editor is using the correct Python virtual environment.

Modify or add SQLModel models for data and SQL tables in ./backend/app/models.py, API endpoints in ./backend/app/api/, CRUD (Create, Read, Update, Delete) utils in ./backend/app/crud.py.

VS Code

There are already configurations in place to run the backend through the VS Code debugger, so that you can use breakpoints, pause and explore variables, etc.

The setup is also already configured so you can run the tests through the VS Code Python tests tab.

Docker Compose Override

During development, you can change Docker Compose settings that will only affect the local development environment in the file docker-compose.override.yml.

The changes to that file only affect the local development environment, not the production environment. So, you can add "temporary" changes that help the development workflow.

Live reload during development is provided by the docker-compose.dev.yml file, enabled with the --dev flag of scripts/compose.sh. It mounts the source as a Docker "host volume" and runs both the backend and the selected workers under a file watcher, so the processes restart whenever you change the code - no image rebuild per change. It also builds the backend with its dev dependencies. This is only for development; for production you build the images with a recent version of the code.

To start the stack with live reload run:

$ ./scripts/compose.sh --dev --workers vector,point-cloud -- up --build

The first run (or after changing dependencies) needs --build so the images include the dev dependencies, in particular the watchfiles watcher. On hosts where file changes are not detected across the mounted volume (Docker Desktop on macOS/Windows, or a repo under /mnt/c in WSL2) set WATCHFILES_FORCE_POLLING=true in your .env.

The backend runs /start-reload.sh (a single auto-reloading server process instead of the multiple production ones). If you save a file with a syntax error the process exits; fix it and the watcher restarts it automatically.

To get inside the running container with a bash session you can then exec inside it:

$ docker compose exec backend bash

You should see an output like:

root@7f2607af31c3:/app#

that means that you are in a bash session inside your container, as a root user, under the /app directory, this directory has another directory called "app" inside, that's where your code lives inside the container: /app/app.

Backend tests

To test the backend run:

$ bash ./scripts/test.sh

To build the backend image with the dev dependencies, bring up a clean stack and run the tests against it locally, use:

$ bash ./scripts/test-local.sh

The tests run with Pytest, modify and add tests to ./backend/app/tests/.

If you use GitHub Actions the tests will run automatically.

Test running stack

If your stack is already up and was built with the dev dependencies (i.e. started with --dev), you can run the tests against it with:

docker compose exec backend bash /app/tests-start.sh

A plain stack (without --dev) is built without the dev dependencies, so pytest is not available in the backend container - use scripts/test-local.sh instead, which builds a test image for you.

That /app/tests-start.sh script just calls pytest after making sure that the rest of the stack is running. If you need to pass extra arguments to pytest, you can pass them to that command and they will be forwarded.

For example, to stop on first error:

docker compose exec backend bash /app/tests-start.sh -x

Test Coverage

When the tests are run, a file htmlcov/index.html is generated, you can open it in your browser to see the coverage of the tests.

Migrations

As during local development your app directory is mounted as a volume inside the container, you can also run the migrations with alembic commands inside the container and the migration code will be in your app directory (instead of being only inside the container). So you can add it to your git repository.

Make sure you create a "revision" of your models and that you "upgrade" your database with that revision every time you change them. As this is what will update the tables in your database. Otherwise, your application will have errors.

  • Start an interactive session in the backend container:
$ docker compose exec backend bash

  • Alembic is already configured to import your SQLModel models from ./backend/app/models.py.

  • After changing a model (for example, adding a column), inside the container, create a revision, e.g.:

$ alembic revision --autogenerate -m "Add column last_name to User model"

  • Commit to the git repository the files generated in the alembic directory.

  • After creating the revision, run the migration in the database (this is what will actually change the database):

$ alembic upgrade head

If you don't want to use migrations at all, uncomment the lines in the file at ./backend/app/core/db.py that end in:

SQLModel.metadata.create_all(engine)

and comment the line in the file prestart.sh that contains:

$ alembic upgrade head

If you don't want to start with the default models and want to remove them / modify them, from the beginning, without having any previous revision, you can remove the revision files (.py Python files) under ./backend/app/alembic/versions/. And then create a first migration as described above.