Upgrade from GeoNode 4 to 5 - GeoNode/geonode GitHub Wiki

BREAKING CHANGE: Redis Migration

In GeoNode 5, Redis has been migrated replacing the RabbitMQ broker and the django-celery-results. The corresponding pull request is available here: https://github.com/GeoNode/geonode/pull/13438

Replacement of the django-celery-results

On GeoNode 4, the django-celery-results library was used by Celery as the backend system, and by GeoNode itself to track the asynchronous tasks of the importer. Now, Redis is used as the Celery backend, but we still needed a way to track the status of the importer’s asynchronous tasks internally. Therefore, in GeoNode 5, a new system was developed for managing the importer’s asynchronous tasks, which uses the ExecutionRequest model, a build-in model of GeoNode.

More specifically, each step of the importer’s processes including the Upload, Copy, Replace and Upsert, is tracked by this model and the user is able to inspect the payload of the ExecutionRequest object during an importer’s workflow. For example in a successful upload process, the payload includes the following :

"tasks": {
    "airports": {
        "geonode.upload.import_resource": "SUCCESS",
        "geonode.upload.publish_resource": "SUCCESS",
        "geonode.upload.create_geonode_resource": "SUCCESS"
        }
    },

where the tasks is the current field in the Execution request object, airports is the name of the dataset while the import_resource, publish_resource and create_geonode_resource are the main steps of an Upload workflow. The status can be SUCCESS or FAILED accordingly.

Replacement of RabbitMQ to Redis

On GeoNode 5, Redis is the default message broker for the asyncronous workflows. This means that for fresh GeoNode 5 installations, Redis will be used by default instead of RabbitMQ, used by GeoNode 4.

However, for a GeoNode 5 installation in an already installed GeoNode 4, without using RabbitMQ (only Redis usage), a few steps should be followed:

  • In the .env file replace this line:
BROKER_URL=amqp://guest:guest@rabbitmq:5672/

with:

BROKER_URL=redis://redis:6379/0
CELERY_RESULT_BACKEND=redis://redis:6379/1

Then, after the build of GeoNode 5 (by using docker compose build), delete the orphaned containers:

cd /path/to/geonode
docker compose down --remove-orphans

After the above, when the users create the containers (by running docker compose up -d) RabbitMQ will not present in the created containers.

Optional RabbitMQ support

On GeoNode 5, RabbitMQ broker was replaced by Redis. However, with a few adjustments in the settings, RabbitMQ can still be supported on GeoNode 5. In this configuration, GeoNode uses Redis as a Celery backend and RabbitMQ as a broker. The steps for the support of RabbitMQ are presented below:

  • In the .env_dev, .env_test and .env_local, replace this line:
BROKER_URL=redis://localhost:6379/0

with:

BROKER_URL=amqp://guest:guest@localhost:5672//
  • In the settings.py replace this line:
REDIS_SIGNALS_BROKER_URL = os.environ.get("BROKER_URL", "redis://localhost:6379/0")

with:

RABBITMQ_SIGNALS_BROKER_URL = os.environ.get("BROKER_URL", "amqp://localhost:5672")

and in the same file this line:

_BROKER_URL = REDIS_SIGNALS_BROKER_URL if ASYNC_SIGNALS else LOCAL_SIGNALS_BROKER_URL

with:

_BROKER_URL = RABBITMQ_SIGNALS_BROKER_URL if ASYNC_SIGNALS else LOCAL_SIGNALS_BROKER_URL

If another broker (e.g Redis) was used before RabbitMQ, delete the default queue before starting Celery:

sudo rabbitmqctl delete_queue default
  • In the docker-compose.yml, docker-compose-dev.yml and docker-compose-test.yml add the RabbitMQ service:
rabbitmq:
    image: rabbitmq:3-alpine
    container_name: rabbitmq4${COMPOSE_PROJECT_NAME}
    volumes:
      - rabbitmq:/var/lib/rabbitmq
    restart: unless-stopped

and in the same file, in volumes add:

rabbitmq:
    name: ${COMPOSE_PROJECT_NAME}-rabbitmq

Note: We should not remove the Redis service and its volume since it is used as Celery backend.

  • In the .env.sample file, replace this line:
BROKER_URL=redis://redis:6379/0

with:

BROKER_URL=amqp://guest:guest@rabbitmq:5672/

Note: For local GeoNode installation (without Docker), RabbitMQ should be installed in the system before running GeoNode. For more information please see: https://www.rabbitmq.com/docs/download#open-source-rabbitmq-server

  • (Formats configurations...)

Importer

  • The importer has been merged inside GeoNode core, For geonode-projects the depencency with the geonode-importer must be removed

Update custom Handlers configuration to work with GeoNode 5

With the issue #12657 a new configuration for serving the configuration for the client is changed, this required some refactor on the handlers in detail:

  • The internal ACTIONS attribute of the handler (the one which lists the tasks to be followed) is renamed into TASKS
  • The upload endpoint now requires an action parameter (previously source) which is used to get the tasks list
  • The source parameter of the ExecutionRequest is dropped
  • If the handler can manage the replace or overwrite_existing_data the key replace must be available in the TASKS list
  • The IMPORT key in the TASKS has been renamed into UPLOAD

So the expecting tasks list for an handler is something similar to this:

TASKS = {
    exa.UPLOAD.value: (
        "start_import",
        "geonode.upload.import_resource",
        "geonode.upload.publish_resource",
        "geonode.upload.create_geonode_resource",
    ),
    exa.COPY.value: (
        "start_copy",
        "geonode.upload.copy_dynamic_model",
        "geonode.upload.copy_geonode_data_table",
        "geonode.upload.publish_resource",
        "geonode.upload.copy_geonode_resource",
    ),
    ira.ROLLBACK.value: (
        "start_rollback",
        "geonode.upload.rollback",
    ),
    ira.REPLACE.value: (
        "start_import",
        "geonode.upload.import_resource",
        "geonode.upload.publish_resource",
        "geonode.upload.create_geonode_resource",
    ),
}
  • The supported_file_extension_config now expects a new format which must be follow. The key ext and optional are dropped
  • New key named formats has been added, is a list of dictionary with three keys: label (the label to be displayed in UI), required_ext which rappresent the mandatory extension for the FE and optional_ext.
  • New actions key is added, contains all the ACTIONS available for the selected handler
  • format has been renamed into type For example:

Before:

@property
def supported_file_extension_config(self):
    return {
        "id": "geojson",
        "label": "GeoJSON",
        "format": "vector",
        "ext": ["json", "geojson"],
        "optional": ["xml", "sld"],
    }

After:

@property
def supported_file_extension_config(self):
    return {
        "id": "geojson",
        "formats": [
            {
                "label": "GeoJSON",
                "required_ext": ["geojson"],
                "optional_ext": ["sld", "xml"],
            },
            {
                "label": "GeoJSON",
                "required_ext": ["json"],
                "optional_ext": ["sld", "xml"],
            },
        ],
        "actions": list(self.TASKS.keys()),
        "type": "vector",
    }