Datastore Migrations - OpenSlides/OpenSlides GitHub Wiki

Assumptions:

  1. Migrations can only be done when the DS is offline, so the migration do not need locking the Datastore.
  2. All events are migrated on a per-position basis, so all data is always up-to-date. Therefore the history is always migrated, too.
  3. Migrations cannot be undone

Restrictions:

  • All data has to be fully migrated - partial migrations are not permitted.

Changes to the Writer:

  • Add integer column version to a new table internals (or similar). This is used to determine which migrations to run.

How does the migrations work:

  • A new module migrations in the Datastore. It builds a framework around the Datastore:
    • Sets up an environment to execute migrations
    • The client code (probably located in the backend) can register a function (class, ...) for each version. Must be sequential from 1 to n.
    • Has a function to execute the migrations. It will execute the migrations in order with a higher version as the current DS version.
  • Each migration either modifies the given events or returns new events which are then added to the same position.
  • After running a migrations, write the new version to the DB.
  • It must be ensured, that calculated tables (e.g., Datastore's models table) are updated.

Workflow of a single migration:

  • For each position, iterate over all events. For each event, the migration function is called an can execute code depending on the event type and content (see below for some general examples)
  • The migration can modify the given event and also create new events
  • The migration has access to:
    • the old data, to check what was previously written in the DB
    • the new data (migrated up to position x-1) to check what was already changed by this migrations in earlier positions

Types of migrations:

  • Add optional field
    • Nothing to do: Field is None in database and handled correctly by the backend
  • Add required field
    • On Create/Update: Add field with correct default value to event
    • On Delete: Nothing to do
  • Add optional relation
    • Nothing to do
  • Add required relation, e.g. for each topic an agenda item is created
    • On Create: Add a new create event for the agenda item and set the relations correctly
    • On Delete: Depends on OnDelete setting, either delete the related model with it or leave it be
  • Remove field
    • On Create/Update: Remove the field from the event
  • Rename field
    • On Create/Update: Rename field in event
  • New permission
    • Nothing to do
  • New permission (assign based on other permission)
    • On Create/Update of group: Check if group should have the new permission. If so, add the new permission to the list.
  • Change field value based on current value
    • On Create/Update: Calculate new value based on current value and set it in event
  • New collection
    • Nothing to do per se, for potential relation updates see above
⚠️ **GitHub.com Fallback** ⚠️