This document discusses the goals and design decisions for WeeWX Version 5.

Complete WeeWX install, including dependencies, using pip, without requiring root privileges. However, the operating system may require privileges in order to install daemon files and to access hardware.

Future-proof the install. Right now, we depend heavily on custom commands in setup.py. However, the imperative approach of setup.py is being phased out in favor of a more declarative approach, using pyproject.toml (see PEP 621 and PEP 631). As an alternative, we could use setup.cfg, but it's clear that pyproject.toml is the future.

Version 5 will require support for importing resources, which was introduced in Python 3.7. However, a backport of importlib.resources is available for Python 3.6. Hence, the earliest supported version of Python will be 3.6 (introduced Dec. 2016). Version 5 will use f-strings, so Python 3.5 cannot be supported.

Support for Python 2.7 will be dropped. All references to six and other compatibility shims will be removed. All __future__ imports will be removed.

Clean separation of WeeWX source code from station data:

• Easier to back up.
• Source code can be considered read-only. It could potentially be distributed as a zip file or Python egg.
• Multiple stations can share the same source code. They just have different station data areas.

See the section Location of user Python code below for details.

The old distutils approach used by previous versions of WeeWX supported a data_files option that could be used to install miscellaneous resources, such as documentation or configuration files, into a final installation target such as /home/weewx.

However, option data_files is being deprecated for a variety of reasons (see this post for a summary). Instead, data resources must now be included as "package data", that is, as part of a Python package. Like any other package, package data includes an __init__.py file, but unlike regular packages, it can include non-code resources.

That means, once installed, these non-code resources will live deep in the target site-packages directory, where they are difficult to find and even more difficult to modify.

Furthermore, package data must be regarded as "read only". For example, it is possible that WeeWX could be installed as a zipfile or Python egg, so, even if a user was willing to find the data resources deep in site-packages somewhere, s/he still could not edit them.

These are the non-code resources that are used in WeeWX and where they will be located as package data:

Previously, we used /home/weewx as the destination for station data, but it required root access in order to set up. With Version 5, the default area becomes ~/weewx-data in the user's home directory. However, /home/weewx can continue to be used by older installations.

Because pip no longer allows free access to the operating system, these user data cannot be set up by pip. Instead, they must be copied into place using a new tool, weectl.

All of this means that installing WeeWX becomes a two-step process:

• Install the source code and package data using pip into standard Python library locations (usually, site-packages).

• Copy package data out of the Python library to a "station data" area using weectl.

See the companion document pip install strategies for all the various way pip and its allies can be used to install WeeWX.

With Version 5, the default area for station data is ~/weewx-data, which allows WeeWX to be installed entirely without root privileges.

The directory /home/weewx served this purpose in previous versions of WeeWX and, indeed, it can continue to be used by just specifying the location /home/weewx/weewx.conf when using the tool weectl. Note that in the past, /home/weewx also held WeeWX code in the bin subdirectory, which, starting with V5, will no longer be used. Instead, all source code will live within the normal Python library structure, except for code used by user-installed extensions, which will live in the station data area.

There are many tools that were considered to manage the Python distribution build process including setuptools, pdm, flit, and poetry. We have chosen poetry. It is mature, simple to use, depends on pyproject.toml, and very popular. Poetry can also be used to do an in situ install, but that's not the intended use case. Most users will use pip.

The ubiquitous setuptools was considered, but poetry is easier to use and its implementation of pyproject.toml is more mature. NB: this only affects the distribution build process. Either way, pip is used to install.

Applications will be invoked using an entry point. This is a small shim, installed by pip, which invokes a function within a module or package.

Most packages used by WeeWX already start with the prefix wee_, which is a reasonable claim to a namespace. However, there are two problem areas: user and schemas. Renaming them would break a lot of code, particularly the renaming of user.

However, the recommended install method is to install into a virtual environment where WeeWX does not have to compete with other Python installs for a unique namespace. A collision is improbable.

Through the years, WeeWX has built up an extensive ecosystem of 3rd party extensions. It is essential that we not lose backwards compatibility with them. For this reason, they should continue to be installed directly into the target installation directories, and not indirectly through package resources.

All extensions will be installed into the station data area. That is, into ~/weewx-data, or /etc/weewx.

This results in the following list of nominal locations, relative to WEEWX_ROOT:

Presently, we have 7 different utilities. Version 5 will combining them into one command called, simply, weectl, with different subcommands. By having a single application, we make it easier to find the appropriate subcommand: the user can simply run weectl --help.

Version 5 will offer these subcommands:

# Create a new station data area in ~/weewx-data (the default)
weectl station create
# Upgrade config file, docs, examples, and daemon utility files in ~/weewx-data.
# Skins will be untouched
weectl station upgrade
# Upgrade the config file only
weectl statioin upgrade --what config
# Upgrade the skins only
weectl station upgrade --what skins

# Install the Windy extension:
weectl extension install https://github.com/matthewwall/weewx-windy/archive/master.zip

Previously, user Python code lived amongst the rest of WeeWX's source code, that is, either under /home/weewx/bin, or /usr/share/weewx. However, with Version 5, we are seeking a clean separation between WeeWX source code and station data. Because user Python code is customized, it will live with other station data. For a pip install, its location will be ~/weewx-data/bin/user for a pip install, it's /etc/weewx/bin/user.

In both caes, note that it's the path of the parent directory that gets injected into PYTHON_PATH when weewxd starts up. This is because user is actually a Python package. We need to find user.foo, not just foo.

As far as possible, we want the configuration file weewx.conf used by package installs to be the same as the file used by pip installs. This will simplify the install logic, as well as support burden.

For this reason, it would be desirable if $USER_ROOT is the same for the two install methods. What is different, is the value for $WEEWX_ROOT.

No ambiguity. For legacy installs, it is tempting to inject both the legacy location (e.g., /usr/share/weewx/) and the new location (/etc/weewx/) into PYTHONPATH, which would make old user code accessible in a new install. However, that tempation should be resisted. Only one, well-defined, location should be used, otherwise there can be inconsistent versions between the two locations, leading to difficult-to-diagnose bugs.

• New pip install. Everything under ~/weewx-data.
• Pip install, using legacy (setup.py) station data. Station data is located under /home/weewx. User Python code can be reused if $USER_ROOT is ./bin/user.
• New package install. Everything, except the database, is under /etc/weewx.
• Package install, using legacy station data. Station data is located under /etc/weewx. User legacy Python code is in /usr/share/weewx/user and cannot be reused.
• Run from git repository.

The table below summarizes the methods and their locations. A question mark means there are several choices.

The following has the pros and cons for different values of $USER_ROOT. It only discusses legacy installs --- new installs will work with any solution.

Pip install, using legacy user python code, would require no changes.

Package install. Cannot reuse legacy python code. All user extensions would have to be either reinstalled, or copied from /usr/share/weewx/user to the new, appropriate $USER_DIR location, perhaps by the postinst script.

Has the advantage that the lib clearly indicates the role of its underlying contents: it's part of a library (not some kind of binary). It is also a little more future-proof --- we have clearly deliniated where user supplied library-related code goes ---- although at the moment it's difficult to see where that might be useful.

Pip install would require either a symbolic link from $WEEWX_ROOT/bin to $WEEWX_ROOT/lib, or the user would have to copy files over to its new location.

Package install. The comments for $WEEWX_ROOT/bin/user above apply, except the contents would have to be copied to /etc/weewx/lib/user.

One other comment on using lib instead of bin. Presently, the prototype user subdirectory is located under bin in the source tree, yet it would finally end up under lib. That's an inconsistency. We could relocate it to a new lib subdirectory, but then what of the rest of the WeeWX code? Why is it under bin, instead of also under lib?

With this approach, the user package would live directly in $WEEWX_ROOT. It is less verbose, but is, perhaps, a bit less future-proof.

Pip installs would require that the user subdirectory be moved:

mv /home/weewx/bin/user/ /home/weewx

This isn't so bad because once the directory user had been moved, /home/weewx/bin would no longer be needed and can be safely deleted.

Package install. The comments for $WEEWX_ROOT/bin/user above apply, except the contents would have to be copied to /etc/weewx/user.

Throughout this document, the following definitions are used