Notes on integrating swupdate into builds for Jetson platforms.

swupdate background

swupdate is a lightweight update framework for embedded systems.

Features:

• Open source (mixed, primarily GPL-2.0)
• Written in C
• Many build-time configuration options to support different features
• Configurable for multiple update methods, partition layouts, etc.
• Flexible contents for update packages
• Signed update packages with content verification; encrypted contents supported as well
• Pre- and post-install (shell) scripts supported, as well as scripts (Lua or shell) to run during the installation
• Embedded Lua interpreter for extending functionality
• IPC interface (using AF_UNIX sockets) for control and progress reporting
• Support for delta updates using rdiff

As a framework, swupdate provides components that can be used to implement an update system, but isn't really ready to use "out of the box." It is general enough to support full A/B updates (called "dual-copy" in the documentation), a single rootfs with an update/recovery image, or A/B combined with a fallback recovery/rescue image. It can also support combined updates of a main system with subsystems (having their own CPUs) in a single package, with the swupdate instance on the main CPU forwarding update packages for installation by swupdate instances on the subsystem CPUs. There is also support for handling updates of microcontrollers connected by UARTs, various types of flash devices, formatting/partitioning storage devices, and other features that make swupdate general enough to support a wide variety of embedded systems.

Streamed updates are supported by enabling install-directly for an image in the package control file (called sw-description). I did not test delta update (rdiff) support.

Signed update packages are supported (using RSA keys or CMS certificates), and package contents can also be encrypted, if desired. When signing support is enabled, the public key or cert/CA cert must be provided when swupdate is run. Similar to Mender artifacts, only the control file is actually signed; the control file includes SHA256 checksums for each of the images/files in the update package and those checksums are verified as the contents are unpacked/streamed from the package.

swupdate can be configured to include an HTTP server (called mongoose) that can be used to upload updates to the device. It can also be configured to include a client (called suricatta) that communicates with a back-end update service (hawkBit by default). These subsystems are compiled into the same binary as swupdate but are forked off as subprocesses at runtime. swupdate can also be configured with a generic HTTP downloader.

When run as a service, swupdate provides IPC interfaces that external programs can use to execute updates (with the update package supplied by the program and sent via the IPC channel) and collect update progress information.

Integration experiments

• Implemented in tegra-test-distro, a fork of the OE4T demo distro.
• Experimented with t186 (TX2) and t194 (Xavier) machines only.
• Implemented "dual-copy" (A/B) updates only.

I used custom flash layouts for each platform tested, identical to the layouts I have used for RAUC and Mender integration testing. I was able to create builds that supported swupdate, Mender, and RAUC in parallel within a single image.

Bootloader integration - t186/t194

swupdate only has built-in support for setting/getting bootloader variables for U-Boot and EFI Boot Guard. There is no provision for other bootloaders, although in its most basic mode of operation, such variable setting is not strictly required. However, its suricatta hawkBit interface expects to be able to set bootloader variables to communicate the update state externally. (Although it is up to the implementer to supply the current state via a command-line parameter when starting swupdate, a fact that isn't clearly documented.)

Handling bootloader updates

Similar to RAUC and Mender integration, I added a post-install script to run tegra-bootloader-update on the BUP bundled into the rootfs. Since tegra-bootloader-update automatically modifies the boot slot on a successful update, no additional changes were needed to implement the logic for updating the boot slot.

Build integration

The swupdate maintainer also provides a meta-swupdate layer for OE/Yocto builds. This layer can be added to the layer set without modifying your existing builds; swupdate itself is largely self-contained, as it is up to the distro/project implementer to handle its integration as part an overall installation/update solution.

To adapt meta-swupdate to our builds, I added a dynamic layer under meta-tegra-support to add config fragments (swupdate uses Kbuild) to select features suitable for the tegra platforms and to add the custom-bootloader patch and an interface script for tegra-bootinfo variable access.

In the meta-testdistro layer, I added another bbappend for the swupdate recipe to enable features specific to my distro and for my testing:

• systemd support
• signed images (along with a bbclass for signing using digsigserver)
• disabling the HTTP server
• enabling the hawkBit interface
• shell script fragments to set the command line parameters to pass to swupdate as part of its startup script.

I also created demo-swupdate-* recipes to create swupdate packages for the demo images included in the distro, and updated the packagegroup-demo-base recipe to include swupdate, swupdate-client, swupdate-progress, and swupdate-tools-hawkbit in all of the demo images.

A/B implementation

The swupdate documentation suggests how to implement a dual-rootfs, A/B update system through the use of its -e (--select) command line option for selecting which "software collection" to install in an update package. The selection consists of two elements, software and mode. I used system as the software element, to generically refer to a complete system update, and use slot_a/slot_b as the mode to indicate the slot from which the system was booted (not the slot to install into, as is used in the examples in the swupdate documentation, which I found to be more confusing when running swupdate as a service at boot time). The control file in the demo image update packages includes a stanza for each of the two modes which directs the installation of the rootfs image into the inactive flash partition (APP_b when running from slot A, APP when running from slot B). Each mode also includes a stanza to configure the post-install script for performing the bootloader update.

The startup script fragment added to swupdate adds the correct -e parameters based on the current boot slot.

hawkBit support

I used the hawkBit Docker image to run an instance of hawkBit on my development machine and tested swupdate with it. It worked more or less as described in the documentation. I added another startup script fragment to set the additional arguments needed, using the machine ID as the hawkBit "controller ID" that uniquely identifies a client, adding the target token (which I manually installed into a file on the target device after enrolling the it via hawkBit's management UI), and passing the -c option to indicate the current update state based on the value of the ustate boot variable, so that after an update and reboot, a success or failure message would be sent to the hawkBit server.

General server support

As an alternative to hawkBit, swupdate also supports communicating with a "general" HTTP server for checking available updates. The protocol used is very simple, and it was easy to put together a basic server to test the protocol used.

Delta update support

Support for delta updates was recently (as of Jan 2022) added to the code base. With some patches to the zchunk package and to swupdate itself, I was able to get get delta updates working. It worked reliably, and the amount of data needing to be downloaded from the server was greatly reduced, as expected, for small updates. Installation time (copying data from one partition to the other, interleaved with the downloaded difference chunks) was not unreasonable, taking about 3 minutes to install a 5.8GB ext4 filesystem image. This is about the same amount of time it took for installing a full image (using compressed ext4, about 2GB compressed for the 5.8GB filesystem) over a LAN. For cloud-hosted updates, however, the reduced network traffic would be a big win.

Performance

I did not run extensive performance tests, but in general, updates using compressed ext4 filesystem images felt slower than with either Mender or RAUC (which also use that format). Switching to a compressed tarball and formatting the targeted partition improved installation time to be comparable to Mender for a 5.8GB ext4 filesystem image. Artifact sizes were also comparable, coming in at just over 2GB for both swupdate and Mender.

swupdate appears to have correctness and a small footprint as primary design goals, rather than high performance. Update progress reporting suffers from misleading indications and long pauses, similar to both RAUC and Mender, for the relatively large rootfs updates we have on the Tegra platforms.

Concluding notes

While getting swupdate installed was relatively simple, fuller integration to implement a basic A/B update mechanism was a bit more difficult than with RAUC. It required several iterations of going through the documentation (which does not appear to be on-line; instead you generate your own copy from the doc directory in the source repository), the meta-swupdate recipes and bbclass files, and combing through the source code and examples in the source repo. The specifics of the sw-description file are particularly confusing, since the configuration language is general enough to cover a wide variety of use cases. The tools are there to provide the basis for an update system, but as the documentation clearly states up front, it is up to the developer/integrator to design that update system based on the specific needs of the project/product.

After that steep learning curve, however, swupdate is very flexible and performs its basic functions reliably.