Python 3 migration guide for extensions - ckan/ckan GitHub Wiki

This describes migrating CKAN extensions to Python 3, to be paired with the forthcoming release of CKAN 2.9, as Python 2 went EOL on 31st December 2019.

Note: this page is a work in progress, it will get updated as CKAN 2.9 is completed and we pool info on getting extensions running. Please contribute.

This table lists most commonly used extensions and their status regarding the Python 3 migration. You can check the already migrated ones for examples:

Extension	Python 3 support	Python 3 PR by @smotornyuk
ckanext-scheming	:heavy_check_mark:	-
ckanext-harvest	:heavy_check_mark:	-
ckanext-dcat	:heavy_check_mark:	-
ckanext-xloader	:heavy_check_mark:
ckanext-spatial	:heavy_check_mark: :
ckanext-pages	:heavy_check_mark:
ckanext-geoview	:heavy_check_mark:
ckanext-googleanalytics	:heavy_check_mark:
ckanext-qa	:disappointed:
ckanext-showcase	:heavy_check_mark:
ckanext-pdfview	:heavy_check_mark:
ckanext-hierarchy	:heavy_check_mark:
ckanext-archiver	:heavy_check_mark:	#73
ckanext-fluent	:heavy_check_mark:

Contents

• Issue template
• Code modernization
• Python 3 preparation
  • Create a separate WIP branch
  • Create mixin plugin implementations for Pylons / Flask
  • Command line commands
  • Web assets (JS and CSS)
  • Templates
  • Test suite
  • Running tests for Python 2 and 3 on Travis
  • Fix Python 3 issues / futurize code

Issue template

You can copy this template in a new issue in the extension repository that you are migrating to help you keep track of the different steps. Note that on some extensions most of the heavy lifting might have already been done by @smotornyuk!

Also note that not all might be relevant to your extension.

The goal is to support running this extension against CKAN 2.9 in both a Python 2 and Python 3 environments.
Please refer to the main documentation for [Python 3 extensions migration](https://github.com/ckan/ckan/wiki/Python-3-migration-guide-for-extensions).

Tasks:

- [ ] Code modernization ([docs](https://github.com/ckan/ckan/wiki/Python-3-migration-guide-for-extensions#code-modernization))
  * [ ] Convert controllers to flask views/blueprints ([docs](https://github.com/ckan/ckan/wiki/Python-3-migration-guide-for-extensions#convert-controllers-to-flask-viewsblueprints))
  * [ ] Import from core ckan using toolkit ([docs](https://github.com/ckan/ckan/wiki/Python-3-migration-guide-for-extensions#import-from-core-ckan-using-toolkit))
  * [ ] Dependencies must be python 3 compatible ([docs](https://github.com/ckan/ckan/wiki/Python-3-migration-guide-for-extensions#dependencies-must-be-python-3-compatible))
  * [ ] Update Travis CI config ([docs](https://github.com/ckan/ckan/wiki/Python-3-migration-guide-for-extensions#update-travis-ci-config))
  * [ ] Compatibility table ([docs](https://github.com/ckan/ckan/wiki/Python-3-migration-guide-for-extensions#compatibility-table))
  * [ ] Release modernized version ([docs](https://github.com/ckan/ckan/wiki/Python-3-migration-guide-for-extensions#release-modernized-version))
- [ ] Python 3 preparation ([docs](https://github.com/ckan/ckan/wiki/Python-3-migration-guide-for-extensions#python-3-preparation))
  * [ ] Create a separate WIP branch ([docs](https://github.com/ckan/ckan/wiki/Python-3-migration-guide-for-extensions#create-a-separate-wip-branch))
  * [ ] Create mixin plugin implementations for Pylons / Flask ([docs](https://github.com/ckan/ckan/wiki/Python-3-migration-guide-for-extensions#create-mixin-plugin-implementations-for-pylons--flask))
  * [ ] Command line commands ([docs](https://github.com/ckan/ckan/wiki/Python-3-migration-guide-for-extensions#command-line-commands))
  * [ ] Web assets ([docs](JS and CSS) [docs](https://github.com/ckan/ckan/wiki/Python-3-migration-guide-for-extensions#web-assets--js-and-css-))
  * [ ] Templates ([docs](https://github.com/ckan/ckan/wiki/Python-3-migration-guide-for-extensions#templates))
  * [ ] Test suite ([docs](https://github.com/ckan/ckan/wiki/Python-3-migration-guide-for-extensions#test-suite))
  * [ ] Running tests for Python 2 and 3 on Travis ([docs](https://github.com/ckan/ckan/wiki/Python-3-migration-guide-for-extensions#running-tests-for-python-2-and-3-on-travis))
  * [ ] Fix Python 3 issues / futurize code ([docs](https://github.com/ckan/ckan/wiki/Python-3-migration-guide-for-extensions#fix-python-3-issues--futurize-code))

Code modernization

Even before you starting the actual Python 3 update, there are often plenty of things to modernize in your extension, which can be done on Python 2 and CKAN 2.8.

Convert controllers to flask views/blueprints

If your extension has any controllers (IMapper + BaseController), they need converting to blueprints (IBlueprint). (Controllers will still work in CKAN 2.9 (py2), but not Python 3. Blueprints are compatible from CKAN 2.7)

Import from core ckan using toolkit

Replace any imports like paste.* and pylons.* with ckan.plugins.toolkit. Use the toolkit whenever possible (or use ckantoolkit if the method or object is not on the core toolkit).

A simple test is to grep for any occurrences of paste or pylons in import statements.

$ git grep -w 'paste\|pylons'

Dependencies must be python 3 compatible

Check if any python libraries that your extension depends on is Python 2 only. Replace if so. You can use caniusepython3 from your virtualenv to identify any blocking dependencies.

$ caniusepython3 -r requirements.txt

Update Travis CI config

Use the Travis config from Running tests for Python 2 and 3 on Travis, but comment out the Python 3 tests for now.

Compatibility table

Add to the README a compatibility table, to be ultra clear about the status of the work to add CKAN 2.9 compatibility.

Release modernized version

Once you've done these modernizations for python 2, it's a good moment to test, merge and release them, before moving onto Python 3 work. This benefits us in the lean tradition of small batch sizes bringing forward downstream work and users finding any issues as early as possible.

Python 3 preparation

These are things we know need to change to run on Python 3 (with the forthcoming CKAN 2.9 release). Remember that to ensure Python 3 compatibility the extension test suite should be run against it, so make sure to cover the last points here.

Aim to keep Python 2 compatibility as well, so existing CKAN 2.8 users have a smoother upgrade path.

Create a separate WIP branch

For instance call it py3. While CKAN itself uses the master branch as dev and new releases turn into branches at some point, it seems that the community expects master branch of extensions to be stable. Nobody wants to get CKAN broken by doing git pull origin master from the extension folder, so don't put unstable code into the master branch.

Create mixin plugin implementations for Pylons / Flask

The goal here is allow conditional loading of version-specific interfaces and methods depending on the CKAN version run. To achieve this, follow this process:

1. Move ckanext/ext/plugin.py into the ckanext/ext/plugin/__init__.py. e.g.:

   mkdir ckanext/xloader/plugin/
   git mv ckanext/xloader/plugin.py ckanext/xloader/plugin/__init__.py
   

   This won't break anything, as the import path for the plugin won't be changed. Why do you need this? Because you'll likely have some logic that should be conditionally applied based on CKAN/Python version. While you can just use six.PY3 and ckan.plugins.toolkit.requires_ckan_version, you'll probably get lost in the woods of conditional imports, method calls, etc. Instead, I suggest to make main plugin(__init__.py) that contains all the shared logic and has only one conditional import - depending on the CKAN version, you'll import either ckanext/ext/plugin/flask_plugin.py or ckanext/ext/plugin/pylons_plugin.py, each of those sub-plugins contains only version-specific logic and imports. See example: https://github.com/ckan/ckanext-harvest/tree/master/ckanext/harvest/plugin But you need to update relative paths, as now file-tree has changed. enable absolute_imports for python modules, and check, whether you registering any new directories(templates, resources, translations). For example, you'll likely make following change:

   p.toolkit.add_template_directory(config, "templates")
   # -> 
   p.toolkit.add_template_directory(config, "../templates")
   

2. Remove the version-dependent code from the original plugin(IMapper/IBlueprint)

3. Create ckanext/ext/plugin/flask_plugin.py and ckanext/ext/plugin/pylons_plugin.py. Each of those files contains version-specific plugin-mixin (just ordinary CKAN plugin). In our case, flask_plugin adds new routes via blueprint, and pylons_plugin is doing it via before_map from IMapper.

4. Depending on current CKAN version, import one of the sub-plugins into main plugin

   import ckan.plugins as p
   
   try:
       p.toolkit.requires_ckan_version("2.9")
   except p.toolkit.CkanVersionException:
       from ckanext.ext.plugin.pylons_plugin import MixinPlugin
   else:
       from ckanext.ext.plugin.flask_plugin import MixinPlugin
   

5. Add version-specific plugin as a mixin to the main plugin class

   class MyPlugin(MixinPlugin, p.SingletonPlugin):
     ...
   

Command line commands

If a plugin registers CLI commands, add ckanext/ext/cli.py with relevant logic.

CKAN>=2.9 uses Click for CLI commands. One can integrate new command into existing ecosystem via IClick interface (new in CKAN 2.9):

class ExtPlugin(p.SingletonPlugin)
         p.implements(p.IClick)
         # IClick
         def get_commands(self):
             """Call me via: `ckan hello`"""
             import click
             @click.command()
             def hello():
                 click.echo('Hello, World!')
             return [hello]

Web assets (JS and CSS)

Create webassets.yml for resources (Documentation

Basically, there are two main cases here. If you were not using resource.config, you probably have some files, like ckanext/ext/fanstatic/script.js and ckanext/ext/fanstatic/style.css. If you are registering fanstatic directories, those files are included inside base.html(or any other template) as following:

{% block scripts %}
  {{ super() }}
  {% resource 'ckanext-ext/script.js' %}
{% endblock scripts %}
{% block styles %}
  {{ super() }}
  {% resource 'ckanext-ext/style.css' %}
{% endblock styles %}

Instead, you need to create ckanext/ext/fanstatic/webassets.yml with the following content:

ext_script:  # name of the asset
  filters: rjsmin # preprocessor for files. Optional
  output: ckanext-ext/extension.js  # actual path, where the generated file will be stored
  contents: # list of files that are included in the asset
    - script.js
ext_style:  # name of asset
  output: ckanext-ext/extension.css  # actual path, where generated file will be stored
  contents: # list of files that are included into asset
    - style.css

And include those assets into the template:

{% block scripts %}
  {{ super() }}
  {% asset 'ckanext-ext/ext_script' %}
{% endblock scripts %}
{% block styles %}
  {{ super() }}
  {% asset 'ckanext-ext/ext_style' %}
{% endblock styles %}

Make sure, that you've created separate groups for js and CSS files - currently, it's not possible to create an asset(bundle) that contains both types. It becomes more relevant if you've used resource.config. In this case, it's probably can look like:

[groups]
ext = script.js
      style.css

And inside the template:

{% block scripts %}
  {{ super() }}
  {% resource 'ckanext-ext/ext' %}
{% endblock scripts %}

In the case of fanstatic, it was possible to mix CSS and js into a single asset(named ext). For webassets you have to split those files into two independent groups(ext_script and ext_style). So, after migration from resource.config to the webassets.yml, you'll still have to update template:

{% block scripts %}
  {{ super() }}
  {% asset 'ckanext-ext/ext_script' %}
{% endblock scripts %}
{% block styles %}
  {{ super() }}
  {% asset 'ckanext-ext/ext_style' %}
{% endblock styles %}

Templates

Optionally, create pylons/flask subfolders for templates if necessary. Or, at least, make sure you are not using any global variables(especially c) in templates and use correct url_for arguments (controller-action for <=2.8, and dot-named routes for newer CKAN versions).

You will receive an error in CKAN <2.9, if you are using an asset tag, as it wasn't registered yet. This can be solved via creating separate snippets for fanstatic's resource (ckanext/ext/templates/ext/snippets/ext_resource.html and ckanext/ext/templates/ext/snippets/ext_asset.html) and then including one of those files depending on CKAN version:

 ```
 {% block scripts %}
    {% set type = 'asset' if h.ckan_version().split('.')[1] | int >= 9 else 'resource' %}
    {% include 'ext/snippets/ext_' ~ type ~ '.html' %}
 {% endblock scripts %}
 ```

The package controller was divided into dataset and resource blueprints. Update link_for and url_for:

 ```
 {# BEFORE #}
 {% link_for _('Datasets'), controller='package', action='search' %}
 {# AFTER #}
 {% link_for _('Datasets'), controller='dataset' if h.ckan_version().split('.')[1] | int >= 9 else 'package', action='search' %}
 ```

Test suite

CKAN>=2.9 uses pytest instead of nose (related PR).

To only have one single test suite, old nose tests need to be rewritten to pytest. The extension needs to install the pytest-ckan plugin so CKAN versions older than 2.9 can access the CKAN pytest fixtures and helpers.

Remove imports of nose.tools. Add: import pytest

Asserts:

assert_equal(x, y) -> assert x == y
eq_(x, y)          -> assert x == y
assert_in(x, y)    -> assert x in y

assert_raises(exc, func, *args)     -> with pytest.raises(exc):
                                           func(*args)

with assert_raises(exc) as cm:      -> with pytest.raises(RuntimeError) as excinfo:
   func(*args)                             func(*args)
assert 'error msg' in cm.exception     assert 'error msg' in str(excinfo.value)

The old FunctionalTestBase is deprecated and should no longer be used. It is replaced by various pytest fixtures. Common things can be done with these decorators, added to test classes or methods:

@pytest.mark.usefixtures(u"clean_db")  # instead of calling helpers.reset_db() or model.repo.rebuild_db()
@pytest.mark.usefixtures(u"clean_index")  # instead of search.clear_all()
@pytest.mark.usefixtures(u"reset_env")  # if you set any environment variables

If you need a plugin, you need these two decorators:

@pytest.mark.ckan_config("ckan.plugins", "test_feed_plugin datastore")  # instead of plugins.load() /unload
@pytest.mark.usefixtures("with_plugins")

Creating fixtures:

@classmethod
def setup_class(self):
    ... 

->

@pytest.fixture(autouse=True)
    setup_test_search_index()
def initial_data(self, clean_db, clean_index):
    ...

Mocking actions:

@helpers.mock_action("datapusher_submit")
def test_submit(mock_datapusher_submit):
    assert not mock_datapusher_submit.called

->

from ckan.logic import _actions

def test_submit(monkeypatch):
    func = mock.Mock()
    monkeypatch.setitem(_actions, 'datapusher_submit', func)
    func.assert_not_called()

Running tests for Python 2 and 3 on Travis

The goal is to end up with a setup like this:

That is:

• Flake 8 stage to flag incompatibilities with Python 3
• pytest tests run on CKAN <= 2.8, on Python 2
• pytest tests run on CKAN >= 2.8, both on Python 2 and Python 3

Here's an example .travis.yml that you can modify to your needs:

language: python
dist: trusty
group: deprecated-2017Q4
services:
    - redis
    - postgresql

install: bash bin/travis-build.bash

script: bash bin/travis-run.bash

stages:
  - Flake8
  - Tests

jobs:
  include:
    - stage: Flake8
      python: 2.7 
      env: FLAKE8=True
      install:
        - pip install flake8==3.5.0
        - pip install pycodestyle==2.3.0
      script:
        - flake8 --version
        # stop the build if there are Python syntax errors or undefined names
        - flake8 . --count --select=E901,E999,F821,F822,F823 --show-source --statistics --exclude ckan
        # exit-zero treats all errors as warnings.  The GitHub editor is 127 chars wide
        - flake8 . --count --max-line-length=127 --statistics --exclude ckan --exit-zero
    - stage: Tests
      python: "2.7"
      env: CKANVERSION=master
    - python: "3.6"
      env: CKANVERSION=master
    - python: "2.7"
      env: CKANVERSION=2.8
    - python: "2.7"
      env: CKANVERSION=2.7
    - python: "2.7"
      env: CKANVERSION=2.6

cache:
  directories:
    - $HOME/.cache/pip

You should then adapt the scripts for building (travis-build.bash) and running (travis-run.bash) to your extension needs.

The more important thing to consider is use the $CKAN_MINOR_VERSION and $PYTHON_MAJOR_VERSION to conditionally execute Python 2 / CKAN <= 2.8 or Python 3 / CKAN >= 2.9 commands. For instance:

if (( $CKAN_MINOR_VERSION >= 9 )) && (( $PYTHON_MAJOR_VERSION == 2 ))
then
    pip install -r requirements-py2.txt
else
    pip install -r requirements.txt
fi

or

if (( $CKAN_MINOR_VERSION >= 9 ))
then
    ckan -c test.ini harvester initdb
else
    paster harvester initdb -c test.ini
fi

Fix Python 3 issues / futurize code

Running the tests on Python 3 will probably make surface different compatibility issues. Common instances are bytes vs str handling, use of basestring, relative imports etc. Depending on the volume of these changes you might prefer to use conditional execution patterns like if six.PY2: or use the future module: pip install future; futurize ckanext -w

Keep at it until all tests are green! :rocket: