[Draft] Upgrade a GeoNode 4 to 5 - GeoNode/geonode GitHub Wiki

Importer

• The importer has been merged inside GeoNode core, For geonode-projects the depencency with the geonode-importer must be removed

Update custom Handlers configuration to work with GeoNode 5

With the issue #12657 a new configuration for serving the configuration for the client is changed, this required some refactor on the handlers in detail:

• The internal ACTIONS attribute of the handler (the one which lists the tasks to be followed) is renamed into TASKS
• The upload endpoint now requires an action parameter (previously source) which is used to get the tasks list
• The source parameter of the ExecutionRequest is dropped
• If the handler can manage the replace or overwrite_existing_data the key replace must be available in the TASKS list
• The IMPORT key in the TASKS has been renamed into UPLOAD

So the expecting tasks list for an handler is something similar to this:

TASKS = {
    exa.UPLOAD.value: (
        "start_import",
        "geonode.upload.import_resource",
        "geonode.upload.publish_resource",
        "geonode.upload.create_geonode_resource",
    ),
    exa.COPY.value: (
        "start_copy",
        "geonode.upload.copy_dynamic_model",
        "geonode.upload.copy_geonode_data_table",
        "geonode.upload.publish_resource",
        "geonode.upload.copy_geonode_resource",
    ),
    ira.ROLLBACK.value: (
        "start_rollback",
        "geonode.upload.rollback",
    ),
    ira.REPLACE.value: (
        "start_import",
        "geonode.upload.import_resource",
        "geonode.upload.publish_resource",
        "geonode.upload.create_geonode_resource",
    ),
}

• The supported_file_extension_config now expects a new format which must be follow. The key ext and optional are dropped
• New key named formats has been added, is a list of dictionary with three keys: label (the label to be displayed in UI), required_ext which rappresent the mandatory extension for the FE and optional_ext.
• New actions key is added, contains all the ACTIONS available for the selected handler
• format has been renamed into type For example:

Before:

@property
def supported_file_extension_config(self):
    return {
        "id": "geojson",
        "label": "GeoJSON",
        "format": "vector",
        "ext": ["json", "geojson"],
        "optional": ["xml", "sld"],
    }

After:

@property
def supported_file_extension_config(self):
    return {
        "id": "geojson",
        "formats": [
            {
                "label": "GeoJSON",
                "required_ext": ["geojson"],
                "optional_ext": ["sld", "xml"],
            },
            {
                "label": "GeoJSON",
                "required_ext": ["json"],
                "optional_ext": ["sld", "xml"],
            },
        ],
        "actions": list(self.TASKS.keys()),
        "type": "vector",
    }

Redis Migration

In GeoNode 5, Redis has been migrated replacing the RabbitMQ broker and the django-celery-results. The corresponding pull request is available here: https://github.com/GeoNode/geonode/pull/13438

Replacement of the django-celery-results

On GeoNode 4, the django-celery-results library was used by Celery as the backend system, and by GeoNode itself to track the asynchronous tasks of the importer. Now, Redis is used as the Celery backend, but we still needed a way to track the status of the importer’s asynchronous tasks internally. Therefore, in GeoNode 5, a new system was developed for managing the importer’s asynchronous tasks, which uses the ExecutionRequest model, a build-in model of GeoNode.

More specifically, each step of the importer’s processes including the Upload, Copy, Replace and Upsert, is tracked by this model and the user is able to inspect the payload of the ExecutionRequest object during an importer’s workflow. For example in a successful upload process, the payload includes the following :

"tasks": {
    "airports": {
        "geonode.upload.import_resource": "SUCCESS",
        "geonode.upload.publish_resource": "SUCCESS",
        "geonode.upload.create_geonode_resource": "SUCCESS"
        }
    },

where the tasks is the current field in the Execution request object, airports is the name of the dataset while the import_resource, publish_resource and create_geonode_resource are the main steps of an Upload workflow. The status can be SUCCESS or FAILED accordingly.

RabbitMQ to Redis broker and optional RabbitMQ support

Moreover, on GeoNode 5, RabbitMQ broker was replaced by Redis. However, with a few adjustments in the settings, RabbitMQ can still be supported on GeoNode 5. In this configuration, GeoNode uses Redis as a Celery backend and RabbitMQ as a broker. The steps for the support of RabbitMQ are presented below:

• In the .env_dev, .env_test and .env_local, replace this line:

BROKER_URL=redis://localhost:6379/0

with:

BROKER_URL=amqp://guest:guest@localhost:5672//

• In the settings.py replace this line:

REDIS_SIGNALS_BROKER_URL = os.environ.get("BROKER_URL", "redis://localhost:6379/0")

with:

RABBITMQ_SIGNALS_BROKER_URL = os.environ.get("BROKER_URL", "amqp://localhost:5672")

and in the same file this line:

_BROKER_URL = REDIS_SIGNALS_BROKER_URL if ASYNC_SIGNALS else LOCAL_SIGNALS_BROKER_URL

with:

_BROKER_URL = RABBITMQ_SIGNALS_BROKER_URL if ASYNC_SIGNALS else LOCAL_SIGNALS_BROKER_URL

• In the docker-compose.yml, docker-compose-dev.yml and docker-compose-test.yml replace this:

redis:
    image: redis:7-alpine
    container_name: redis4${COMPOSE_PROJECT_NAME}
    volumes:
      - redisdata:/data
    restart: unless-stopped

with:

rabbitmq:
    image: rabbitmq:3-alpine
    container_name: rabbitmq4${COMPOSE_PROJECT_NAME}
    volumes:
      - rabbitmq:/var/lib/rabbitmq
    restart: unless-stopped

and this:

redisdata:
    name: ${COMPOSE_PROJECT_NAME}-redisdata

with:

rabbitmq:
    name: ${COMPOSE_PROJECT_NAME}-rabbitmq

• In the .env.sample file, replace this line:

BROKER_URL=redis://redis:6379/0

with:

BROKER_URL=amqp://guest:guest@rabbitmq:5672/

Note: For local GeoNode installation (without Docker), RabbitMQ should be installed in the system before running GeoNode. For more information please see: https://www.rabbitmq.com/docs/download#open-source-rabbitmq-server

• (Formats configurations...)