Using wheels to distribute Python packages - lwfitzgerald/spksrc GitHub Wiki

Wheels

If your SPK uses Python packages you can use the 'wheel' format to distribute the packages together with your SPK. Read up on the format here.

Pure-python or cross-compiling

Generally speaking, there are two types of Python packages:

  • Pure-python packages. Wheels are platform independent, and are quick to create;
  • Packages with (optional) C-extensions. These packages have to be compiled with GCC, and require a (cross-)compiled Python to be available.

For spksrc specifically, we also define a third type of package:

  • Packages with C-extensions, which depend on other cross-packages at build time;
  • Packages that need patches to be able to create a working wheel. This type of package requires a new cross package to be created. Generally speaking, these packages will also require a (cross-)compiled Python to be available.

How does spksrc handle wheels?

Pure-Python packages

Add the package to WHEELS in the SPKs Makefile (either directly, or by using a requirements.txt). In order to create reproducible builds, all the required packages should be frozen to a specific version (e.g. docutils==0.12).

spksrc will store a requirements.txt in $(WORK_DIR)/wheelhouse. Using that requirements file, spksrc will create a pure-python wheel by clearing all the build flags. The original wheel is stored in $(WORK_DIR)/wheelhouse. Finally, spksrc will rename and copy the wheel to $(INSTALL_DIR)/$(INSTALL_PREFIX)/share/wheelhouse for further processing.

Packages with C-extensions

  • Add BUILD_DEPENDS = cross/python cross/setuptools cross/pip cross/wheel to the SPKs Makefile. This ensures that Python is cross-compiled and other requirements are setup correctly to create cross-compiled wheels.
  • Add the package to WHEELS in spk/Makefile (either directly, or by using a requirements.txt). In order to create reproducible builds, all the required packages are frozen to a specific version (e.g. mercurial==4.0.1).

spksrc will store a requirements.txt in $(WORK_DIR)/wheelhouse. It will then cross-compile Python and process python-cc.mk. Using the requirements file, spksrc will create cross-compiled wheel which is stored in $(WORK_DIR)/wheelhouse. Finally, spksrc will rename and copy the wheel to $(INSTALL_DIR)/$(INSTALL_PREFIX)/share/wheelhouse for further processing.

Python packages using a cross package

  • Create a new package in cross/ with the correct details. Add an include to spksrc.python-wheel.mk in the Makefile, so spksrc knows what to do with the package.
  • Add BUILD_DEPENDS = cross/python cross/setuptools cross/pip cross/wheel to the SPKs Makefile. This ensures that Python is cross-compiled and other requirements are setup correctly to create cross-compiled wheels.
  • Also add the new cross package to BUILD_DEPENDS in the SPKs Makefile. In contrast to the other two types, this type of package should not be included in WHEELS or requirements.txt.

spksrc will cross-compile Python, then process python-cc.mk. Due to include ../../mk/spksrc.python-wheel.mk, spksrc creates a cross-compiled wheel. The resulting wheel is stored in $(WORK_DIR)/wheelhouse. Finally, spksrc will rename and copy the wheel to $(INSTALL_DIR)/$(INSTALL_PREFIX)/share/wheelhouse for further processing.

Next steps

After the above, spksrc will resume its normal activities to build the SPK.

  • To include $(INSTALL_DIR)/$(INSTALL_PREFIX)/share/wheelhouse in the SPK itself, add rsc:share/wheelhouse to the SPKs PLIST.
  • In addition, the installer should contain a line to install the wheels in a Python virtualenv. The generic format to create a virtualenv and install wheels for SynoCommunity packages is:
# Create a Python virtualenv
    ${VIRTUALENV} --system-site-packages ${INSTALL_DIR}/env > /dev/null
# Install the wheels
    ${INSTALL_DIR}/env/bin/pip install --no-deps --no-index -U --force-reinstall -f ${INSTALL_DIR}/share/wheelhouse ${INSTALL_DIR}/share/wheelhouse/*.whl > /dev/null 2>&1

Example

Let's look at the Mercurial SPK

Add python wheels

The Mercurial SPK contains two Python packages: Mercurial itself, and Docutils, which is a dependency of Mercurial.

Mercurial needs cross-compiling because it contains C-extensions. In addition, it also has to be patched to ensure a working wheel is created. That means it's a package of the third type as previously described. Docutils on the other hand is a pure-python package.

Starting off with Docutils: Mercurials Makefile sets WHEELS = src/requirements.txt. This requirements file contains docutils==0.12 as its only entry. This is all that needs to be done to create a Docutils wheel.

For Mercurial itself, a bit more is needed:

  • spksrc/cross/mercurial/Makefile is created with the correct content. There's no need for dependencies in this case, as docutils is handled via the requirements file.

  • The Makefile's include is set to create cross-compiled wheels: include ../../mk/spksrc.python-wheel.mk.

  • The appropriate patches for Mercurial are added to the patches directory.

  • A digests file should be created, to ensure the file download is not corrupted.

  • The SPKs Makefile then needs the following: BUILD_DEPENDS = cross/python cross/setuptools cross/pip cross/wheel to cross-compile Mercurial. BUILD_DEPENDS also contains cross/mercurial (although it could also be added to DEPENDS as there's nothing in the PLIST)

  • The last step is to add rsc:share/wheelhouse to the SPKs PLIST.

Building the SPK via make arch-$(ARCH) should now result in two wheels in $(WORK_DIR)/wheelhouse. The wheels are also stored in $(INSTALL_DIR)/$(INSTALL_PREFIX)/share/wheelhouse, but with a generic naming format to ensure the wheels are recognized as valid on the target device. During the processing of the SPKs PLIST, the wheelhouse directory is copied to $(STAGING_DIR)/share/wheelhouse.

Installing the wheels on the target device

Once the Python packages are succesfully created and included in the package, you'll need to make sure the wheels are installed.

  • In Mercurial installer, include the generic command to first create a Python virtual environment: ${VIRTUALENV} --system-site-packages ${INSTALL_DIR}/env > /dev/null. Note that in some cases you might not want to use --system-site-packages.
  • Install all available wheels into the virtual environment as follows: ${INSTALL_DIR}/env/bin/pip install --no-deps --no-index -U --force-reinstall -f ${INSTALL_DIR}/share/wheelhouse ${INSTALL_DIR}/share/wheelhouse/*.whl > /dev/null 2>&1

Tips and tricks

  • Generally speaking, you should start with the assumption that all the required Python packages are pure-python. When building a pure-python wheel fails, the build process will halt with an error, after which you can decide what to do. A good next step is to assume that one or more packages should be cross-compiled, which means adding BUILD_DEPENDS = cross/python cross/setuptools cross/pip cross/wheel and see if that works better.
  • In a number of cases, the wheels name can help you see if a package is pure-python or not. E.g. if maintainers have uploaded wheels to PyPI, pure-python packages generally end with a format like py2-none-any.whl or py2.py3-none-any.whl. Packages with C-extensions might end with cp34-cp34m-manylinux1_x86_64.whl. This also applies to wheels created by spksrc.
  • It is required to pin packages to a specific version. Example: mercurial==4.0.1. Other version specifiers are not allowed.
  • It is required to add all requirements to the package. Upstream maintainers sometimes only list so-called top-level requirements for their packages, and rely on pip to process dependencies during installation. This can cause issues during installation of SynoCommunity packages. To make sure all the requirements are included in your final requirements.txt, run pip install -r requirements.txt (without specifying --no-deps) in a separate virtualenv. After pip has procesed all the requirements, run pip freeze, and use that output as starting point for the final requirements.txt.
  • References to setuptools, pip or wheel should not be included in requirements.txt or be commented out.
  • Python packages that are processed as DEPENDS, or cross packages, should not be included in requirements.txt or be commented out.
  • Errors such as command 'gcc' failed with exit status 1 means cross-compiling is required.
  • In some cases, wheels appear to build successfully as a pure-python wheel, but fail to install or work correctly on the target. Make sure to test the package, and if you run into issues, try to cross-compile the wheel instead.