🚧 Work in Progress 🚧

This is an ongoing effort to document common issues when upgrading extensions from CKAN 2.9 to CKAN 2.10. Feel free to add new items as long as they are not too specific to your particular extension.

All methods from IPackagController and IResourceController were renamed to include the entity name in order to avoid clashes between the two interfaces, eg IPackageController.before_create() -> IPackageController.before_dataset_create() and IResourceController.before_create() -> IResourceController.before_resource_create() (full list of renamed hooks in #6501).

If your CKAN extension just targets one CKAN version just rename the hooks on your plugins so they get call again, eg change this:

class MyPlugin():

    p.implements(p.IPackageController)

    def after_create(self, context, data_dict):
        # Do whatever

to this:

class MyPlugin():

    p.implements(p.IPackageController)

    def after_dataset_create(self, context, data_dict):
        # Do whatever

If your extension targets multiple CKAN versions you can just add both hooks to the plugin class and the relevant one will get called on each CKAN version, eg:

class MyPlugin():

    p.implements(p.IResourceController)

    # CKAN < 2.10
    def after_create(self, context, data_dict):
        return self.after_resource_create(context, data_dict)

    # CKAN >= 2.10
    def after_resource_create(self, context, data_dict):
        # Do whatever

If you are using a custom dataset schema, you might several failures similar to this one:

E           TypeError: str cannot be used as validator because it is not a user-defined function

On CKAN 2.10 you can no longer use built-in Python functions like str(), bool() or the old unicode() as field validators. The most common cause for this is using unicode as a validator for one or more of your fields, for instance if using ckanext-scheming:

    {
      "field_name": "my_custom_field",
      "label": "My custom field",
      "validators": "ignore_missing unicode",
    }

You just need to replace it with the core validator unicode_safe:

    {
      "field_name": "my_custom_field",
      "label": "My custom field",
      "validators": "ignore_missing unicode_safe",
    }

If you are not using ckanext-scheming and are extending the dataset schema directly with IDatasetForm you will have to import the validator with toolkit.get_validator() as usual, eg:

class MyPlugin():

   def create_package_schema(self):
       ignore_missing = tk.get_validator("ignore_missing")                                                                                                                                                                                                                                
       unicode_safe = tk.get_validator("unicode_safe")
       schema = default_create_package_schema()
       schema["my_custom_field"] = [ignore_missing unicode_safe]
       return schema

Note: unicode_safe was introduced in CKAN 2.8. If you need to use it on a previous CKAN version you can import it from the ckantoolkit package (using at least version 0.0.7)

When migrating activities to it's own extension, we changed the trigger for when the activity is created. Starting CKAN 2.10, activities are created using signals after a successful execution of actions (including chaining actions).

This means that a logic in CKAN 2.9 doing the following will no longer work under CKAN 2.10:

@toolkit.chained_action
def package_create(up_func, context, data_dict):
    result = up_func(context, data_dict)
    _custom_method_handling_package_activity(result["id"]) # This will raise Activity not found
    return result

For more information read https://github.com/ckan/ckan/issues/7624.

CKAN 2.10 introduces flask-login for our authorization layer. Please consider visiting the package documentation to get familiar with it's objects and interface.

When dealing with anonymous users CKAN 2.10 introduced the new AnonymousUser object. This will cause the content of auth_user_obj to change from None to AnonymousUser when checking for anonymous users.

From:

# using package_show as example
def package_show(context, data_dict):
    if context.get("auth_user_obj") is not None:
        return {"success": True}
    return {"success": False, "msg": "You must be signed in to view this page"}

To:

def package_show(context, data_dict):
    # from ckan.model import AnonymousUser
    if not isinstance(context["auth_user_obj"], AnonymousUser):
        return {"success": True}
    return {"success": False, "msg": "You must be signed in to view this page"}

Or:

def package_show(context, data_dict):
    user = context["auth_user_obj"]
    if user and not user.is_anonymous:
        return {"success": True}
    return {"success": False, "msg": "You must be signed in to view this page"}

Or:

def package_show(context, data_dict):
    user = context["auth_user_obj"]
    if user and user.is_authenticated:
        return {"success": True}
    return {"success": False, "msg": "You must be signed in to view this page"}

See https://flask-login.readthedocs.io/en/latest/#your-user-class for reference.

CKAN 2.10 only allows 1 email to be registered in the system. Therefore the call User.by_email() now returns .first() instead of .all(). All calls expecting a list should be modified. Example:

From:

    user = model.User.by_email('[email protected]')[0]

To:

    user = model.User.by_email('[email protected]')

To keep compatibility for 2.9 and 2.10:

    user = model.User.by_email('[email protected]')
    if user and isinstance(user, list):
        user = user[0]

In addition, a database migration will be run to ensure there are not 2 active users with the same email in the system. Make sure that this is the case in your database or any ckan db init command will fail with: (psycopg2.errors.UniqueViolation) could not create unique index "idx_only_one_active_email"

If your plugin imports the Activity model class you will get this error:

AttributeError: module 'ckan.model' has no attribute 'Activity'

Activities were moved to a separate plugin in so you need to update the import path:

from ckanext.activity.model import Activity

Or if you need to maintain compatibility with CKAN<=2.9:

try:
    from ckanext.activity.model import Activity
except ImportError:
    from ckan.model import Activity

Starting from CKAN 2.10 all forms submitted from the UI need to include the CSRF token. To include them, just add the following helper call to your form templates:

<form class="dataset-form form-horizontal" method="post" enctype="multipart/form-data">
     {{ h.csrf_input() }}

If your extension needs to support older CKAN versions, use the following:

<form class="dataset-form form-horizontal" method="post" enctype="multipart/form-data">
     {{ h.csrf_input() if 'csrf_input' in h }}

Forms that are submitted via JavaScript modules also need to submit the CSRF token, here’s an example of how to append it to an existing form:

  // Get the csrf value from the page meta tag
  var csrf_value = $('meta[name=_csrf_token]').attr('content')
  // Create the hidden input
  var hidden_csrf_input = $('<input name="_csrf_token" type="hidden" value="'+csrf_value+'">')
  // Insert the hidden input at the beginning of the form
  hidden_csrf_input.prependTo(form)

API calls performed from JavaScript modules from the UI (which use cookie-based authentication) should also include the token, in this case in the X-CSRFToken header. CKAN Modules using the builtin client to perform API calls will have the header added automatically. If you are performing API calls directly from a UI module you will need to add the header yourself.

CKAN 2.10 ships with Bootstrap 5 by default. The custom HTML data attributes used by the Bootstrap JS components (eg modal dialogs, popups, etc) are now prefixed with data-bs- instead of just data-, so you need to update those to trigger the actions:

diff --git a/ckanext/harvest/templates/source/new_source_form.html b/ckanext/harvest/templates/source/new_source_form.html
index 6ea495a4..eda3e201 100644
--- a/ckanext/harvest/templates/source/new_source_form.html
+++ b/ckanext/harvest/templates/source/new_source_form.html
@@ -37,7 +37,7 @@
         <label class="radio">
           <input type="radio" name="source_type" value="{{ harvester['name'] }}" {{ "checked " if checked }} data-module="harvest-type-change">
           {{ harvester['title'] }}
-          <i class="fa fa-question-circle icon-question-sign muted" title="{{ harvester['description'] }}" data-toggle="tooltip"></i>
+          <i class="fa fa-question-circle icon-question-sign muted" title="{{ harvester['description'] }}" data-bs-toggle="tooltip"></i>
         </label>
       {% endfor %}
     </div>

If your plugin needs to support different Bootstrap / CKAN versions you can just add both versions of the data attributes and the other one will be ignored:

diff --git a/ckanext/harvest/templates/source/new_source_form.html b/ckanext/harvest/templates/source/new_source_form.html
index 6ea495a4..eda3e201 100644
--- a/ckanext/harvest/templates/source/new_source_form.html
+++ b/ckanext/harvest/templates/source/new_source_form.html
@@ -37,7 +37,7 @@
         <label class="radio">
           <input type="radio" name="source_type" value="{{ harvester['name'] }}" {{ "checked " if checked }} data-module="harvest-type-change">
           {{ harvester['title'] }}
-          <i class="fa fa-question-circle icon-question-sign muted" title="{{ harvester['description'] }}" data-toggle="tooltip"></i>
+          <i class="fa fa-question-circle icon-question-sign muted" title="{{ harvester['description'] }}" data-bs-toggle="tooltip" data-toggle="tooltip"></i>
         </label>
       {% endfor %}
     </div>

On pytest 7.x (CKAN 2.10), test classes setup() methods (inherited from nose) are not fired or fired before the provided fixtures (thus bypassing db cleaning, etc). It's unclear if this is a regression in pytest or expected behaviour (https://github.com/pytest-dev/pytest/discussions/10005) but in any case it's a good idea to migrate to the recommended pytest way of using fixtures for this.

For instance:

@pytest.mark.usefixtures("clean_db", "clean_index")
class TestSomeSearch:

    def setup(self):

       factories.Dataset()
       factories.Dataset()

    def test_search():

       assert call_action("package_search", q="*:*")["count"] == 2

Can be migrated to:

@pytest.fixture
def some_search_fixtures():
   factories.Dataset()
   factories.Dataset()


@pytest.mark.usefixtures("clean_db", "clean_index", "some_search_fixtures")
class TestSomeSearch:

    def test_search():

       assert call_action("package_search", q="*:*")["count"] == 2

Methods to be run at the end of the test (teardown() in nose) should be run after yielding on the fixture:

@pytest.fixture
def some_search_fixtures():
   # Anything here will be run before running the test
   factories.Dataset()
   factories.Dataset()
   yield
   # Anything here will be run after running the test
   helpers.clean_db()

If you need to share references of the created objects in a fixture across tests you can do:

@pytest.fixture
def some_search_fixtures():
   return {
       "dataset1": factories.Dataset(),
       "dataset2": factories.Dataset()
   }
   # Or just return locals()


@pytest.mark.usefixtures("clean_db", "clean_index")
class TestSomeSearch:

    def test_search(some_search_fixtures):

       assert call_action("package_search", q="*:*")["results"][0]["title"] == some_search_fixtures["dataset1"]["title"]

To avoid triggering CSRF protection failures when sending POST data using the app test fixture, make sure to authenticate via the Authorization header with a valid token rather than using the old REMOTE_USER + user name variant. So turn this:

# CKAN < 2.10
def test_send_some_data(app):
   user = factories.User()
   env = {"REMOTE_USER": user["name"]}

   app.post(url, data=data, environ_overrides=env)

To this:

# CKAN >= 2.10
def test_send_some_data(app):
   user = factories.UserWithToken()
   env = {"Authorization": user["token"]}

   app.post(url, data, environ_overrides=env)

If your extension needs to support multiple CKAN versions, you can use a fixture like the following:

import ckantoolkit


@pytest.fixture
def sysadmin_env():
    try:
        from ckantoolkit.tests.factories import SysadminWithToken
        user = SysadminWithToken()
        return {'Authorization': user['token']}
    except ImportError:
        # ckan <= 2.9
        from ckantoolkit.tests.factories import Sysadmin
        user = Sysadmin()
        return {"REMOTE_USER": user["name"].encode("ascii")}

def test_send_some_data(app, user_env):

   app.post(url, data, environ_overrides=sysadmin_env)

This is a Python 3.9 support issue, but is more likely to surface on recent CKAN versions:

Traceback:
/usr/local/lib/python3.9/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
ckanext/scheming/tests/test_helpers.py:3: in <module>
    from mock import patch, Mock
E   ModuleNotFoundError: No module named 'mock'

You just need to update your mock imports to:

from unittest.mock import patch, Mock

If you need to support multiple Python / CKAN versions you can use:

try:
    from unittest.mock import patch, Mock
except ImportError:
    from mock import patch, Mock