Upping the export version

When upping the export version, one must go through the relevant database migrations, starting from the oldest that has not been considered, sequentially going forward to the newest migration. Starting from these database migrations, one must go through the following:

1. Export/Import archive schema
2. Archive migrations
3. Tests

The last point contains several points, most importantly one must create new export archives under aiida.backends.tests.fixtures.export.migrate. and in the aiida-export-migration-tests repository.

Export/Import archive schema

Update the archive schemes in all_fields_info for each entity, according to the database migrations, if relevant.

Further changes may have to be performed to the general import or export functions, depending on the degree of changes.

Important: If no changes are made to the schema, it may still be relevant to up the export version, since the value of an existing field must be changed. However, if a database migration introduces changes to fields/rows which have never been exported, the schemes may not have to be changed, and there is therefore no reason to up the export version.

Archive migrations

Export archive migrations can be found under aiida.cmdline.utils.migration.. A single file is dedicated to each migration step, e.g., between v0.4 to v0.5.

v0X_to_v0Y.py

There should always be a function called migrate_vX_to_vY(metadata, *args), where X and Y are the old and new export versions, respectively. The function may also require data, if any migrations must be performed on an archive's data.json file. It must have the *args parameter (the linter can be disabled for this particular code line by adding # pylint: disable=unused-argument if necessary).

An implementation of this function may look like:

def migrate_vX_to_vY(metadata, data, *args):  # pylint: disable=unused-argument
    """
    Migration of export files from v0.X to v0.Y

    This is from migration 00?? (some_migration_name) and onwards
    """
    old_version = '0.X'
    new_version = '0.Y'

    verify_metadata_version(metadata, old_version)
    update_metadata(metadata, new_version)

    # Apply migrations
    migration_some_migration_name(metadata, data)
    migration_some_other_migration_name(data)

Here a database migration called some_migration_name and some_other_migration_name are addressed by the Python functions migration_some_migration_name(metadata, data) and migration_some_other_migration_name(data).

__init__.py

In the migration module's __init__.py file is the migrate_recursively function, which takes care of calling the relevant migration functions. One must add the new migrate_vX_to_vY function to the imports as well as to the mapping MIGRATE_FUNCTIONS.

Tests

There are two different testing modules that must be considered:

1. Export/Import tests for changes introduced to export/import
2. Archive migration tests

Export/Import tests

These tests may not be relevant, if changes are only for the schema. However, one should always consider carefully before choosing not to make tests that make sure changes are correctly implemented.

At this moment, tests can be added to test_export_and_import.py in aiida.backends.tests. In general, one should be able to locate the relevant test Class, under which a test can be added. The test Classes are divided into ORM entities and more abstract concepts, such as complex/simple according to the tests involved, which test the general export/import functionality.

Consider carefully whether you cannot use the existing Classes, before creating a new test Class for your tests.

Archive migration tests

A separate GitHub repository (aiida-export-migration-tests) has been created to keep representative export archives for each export version. These were created with an AiiDA version of the time, when the specific export version was active.

The actual tests are still found in the aiida-core repository, under aiida.backends.tests.tools.importexport.migration..

For each new export archive migration, a new test file must be made, e.g., test_v0X_to_v0Y.py, for migrations from v0.X to v0.Y. As a current convention, there are two general tests in each file. The first, test_migrate_vX_to_vY simply tests the migrate_vX_to_vY function, utilizing the simple aiida-core archives under aiida.backends.tests.fixtures.export.migrate.. Secondly, test_migrate_vX_to_vY_complete attempts to test each sub-migration, utilizing the files in the aiida-export-migration-tests repository. While the first test can usually be copy-pasted from previous test files, the latter must be rewritten for each update, in order to correspond to each migration that is performed when upping the export version.

Lastly, one should add the test test_v0X_to_newest in the file test_migration.py. This should be a "small" test, since it should not re-check what is already tested in the test_mgirate_vX_to_vY_complete function. The point is simply to make sure an old export version can always be updated to the newest export version, checking that the specifics in the file are correctly imported after the migration (according to the present export/import schemes).