swupdate project status

For an introduction, see the related Venus issue: #27

Task list

General implementation

• resume https
  • basic implementations works. To resume, the swupdate process must be still running. There is currently a timeout of 30 seconds, and 3 retries. See here.
  • don't start resuming if web-hosted image has been replaced by another one in the mean time (using timestamped url for download)
  • download should time out if network goes down
  • reconsider the retry & resume strategy: downloading ~75MB on a flaky 2G connection might be interrupted more that 3 times. Why not retry forever? Or for 24h for example.
• checking for updates should cost less data then current solution. Easy to accomplish, as opkg update is a quite resource wasting operation. swupdate only downloads first 1000 bytes of .swu file. Where opkg update downloaded ~ 250kB. Every day.
• use https for the transfer
• update gui (git branch)
  • there should be feedback to the user that download of a new version has started
  • add option to only check for now updates, but not install/activate them
  • add option to revert to previous version (= other partition)
• log updating progress, on the data partition. Like we do now to /data/log/opkg/. Done, see /log/swupdate
• keep the four feeds: develop, testing, candidate, release
• don't break the opkg update & install mechanism. Not to be used for normal updates, but to offer extra optional package for users. Like we currently do with qt5, git, gdbserver, etc etc.
• since /home/root will be on /data, move support key files to rootfs. Done: since v1.70 in /usr/share/support-keys/authorized_keys
• check for updates after boot check-update.sh -auto). This was previously done by the gui, but -finally!- not any longer.
• move scripts to separate package (not opkg-scripts) and file system location. Done swupdate-scripts are now here
• devise a mechanism or procedure to test if auto updating still works before releasing an image. Preferably automated, but also ok to move the automated part to the future work section
• add offline updating, which leaves /data/ as is, from an sdcard or usb stick. more info
• check use of the version tag in sw-description file. See mailthread on swupdate mailinglist
• offline updating: make it possible to downgrade, as right now it is only possible to upgrade
• make install script write the installer or current rootfs version (/opt/color-control/version) to /data/installer/version. To always be able to see the version used to create the partitions etc.

CCGX Specific

• use opkg updates to migrate / move all settings that are still on rootfs to data partition

  • ip and wifi keys and settings. Connman has now a hardcoded path to /data/var/lib/connman instead of /var/lib/connman. And updating to v1.70 migrates existing settings to /data
  • gui language. Done: updating to v1.70 migrates existing settings to /data
  • vecan-dbus aka mk2-dbus settings. Done: updating to v1.70 migrates existing settings to /data

• make migration feature, executed during first swupdate install: swupdate script in venus-upgrade-initramfs

  • contents of the /data partition, except /data/log.
  • user configured ssh public keys for root. Note that since v1.70, the home dir for root can be in two locations, depending on used update method (opkg vs recovery)

• release changes that we do want to release to opkg feeds before abandoning them

  • Websockets security update (implemented, part of v1.70)
  • Disable root password (implemented, part of v1.70)
  • Check if there are other security updates that need to be deployed. Limited though, the real solution is to get people to upgrade to swupdate style updates, instead of being abandoned -> Jeroen

• keep existing u-boot on ccgx when installing from SD/USB

• update release instructions & scripts. For current release instructions see install/RELEASE.txt

Beaglebone specific

• make a bootable image from/with the venus-install-sdcard output
• make the leds blink / not blink to show that installing the image is completed and ready for sdcard removal & reboot.
• change rootfs partition size to 320MB: Total available size of bbb is ~1780MB. Minus 128MB data partition is ~1600MB. Divide that by 5 comes to ~320MB. Remaining size on emmc will go into the /scratch partition.
• fix issue that it boots from the sdcard now when a bootable card is insterted. It should look at boot button. Or still boot from sdcard but in u-boot on the sdcard check for boot button and when not pressed boot the active rootfs instead of booting the initramfs image on the sdcard.

Update strategy

There are two kernels and two rootfs-es. One set is active, the other standby. And to update, the active set downloads and streams the new version into the standby partition(s), and then boots into that one.

New partition / volume layout for ccgx/bpp3

dev   size name
--------------------
mtd0  128k spl1
mtd1  128k spl2
mtd2  128k spl3
mtd3  128k spl4
mtd4  1M   u-boot
mtd5  256k env1
mtd6  256k env2
mtd7  1M   u-boot2
mtd8  256k bootparms (unusued)
mtd9  768k splash
mtd10 6M   kernel1
mtd11 6M   kernel2
mtd12 496M ubisystem, split up in three volumes:
           200M rootfs1
           200M rootfs2
           32M  data
           rest is used by ubi for wear leveling etcetera

Usage (on a CCGX running v1.80~2):

# df
Filesystem           1K-blocks      Used Available Use% Mounted on
ubi0:rootfs1            190680     69068    121612  36% /
none                    123144        72    123072   0% /dev
tmpfs                   123144      1708    121436   1% /var/volatile
tmpfs                   123144         0    123144   0% /media/ram
ubi0:data                29032       488     27052   2% /data

Partition / volume layout for the beaglebone

Total available space on a beaglebone black: ~1.78GB

Layout (defined in script here).

8M    boot partition
256M  rootfs1
256M  rootfs2
128M  data partition

Contrary to the ccgx, the kernel and the .dtb file are inside the rootfs in the beaglebone.

Usage (on a beaglebone installed with the sdcard installer):

t# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/root               173679    126001     34828  78% /
devtmpfs                243808         0    243808   0% /dev
tmpfs                   252136       204    251932   0% /run
tmpfs                   252136       256    251880   0% /var/volatile
/dev/mmcblk1p5          122835      2630    111031   2% /data

User documentation to be updated when releasing

• https://github.com/victronenergy/venus/wiki/ccgx-booting-and-buttons
• https://www.victronenergy.com/live/ccgx:firmware_updating
• https://www.victronenergy.com/live/ccgx:ccgx_faq#what_type_of_networking_is_used_by_the_color_control_gx_tcp_and_udp_ports
• https://www.victronenergy.com/live/ccgx:root_access
  • don't use a rootpassword, as password is kept on rootfs and rootfs is replaced entirely during an update: password will reset every update. Instead work with ssh keys, safer anyway. Or disable automatic updates.

New features for in the change log:

• Fixed bugs that bricked the device during firmware update process: no matter when you cut the power, it will always recover. Fixes the 'my Color Control GX only shows the blue logo, what should I do?" complaints
  • Don't rely on the gui to check for updates after bootup
  • Don't rely on the gui to link finding out that there are updates available to actually starting the update
• Updates downloaded via https instead of http
• Reduced data consumption with approx 7.5MB per month. This 7.5MB was caused by the daily check for a new version. Downloading the a new version might increase data consumption though, since now full image is downloaded every time.
• Gui now properly warns that a download and install has started. Also it now warns that the device has initiated a reboot.
• (new) backup image: on the flash, there are now two images. The user can switch back to previous used version without having to get sdcard or internet access as all is still available on ccgx.

Upgrading a ccgx from opkg-install to swupdate-install

Once swupdate is finished and available for download, the devices in the field will receive one last opkg update that shows a message to the user that, for future updates, they need to follow xx instructions: download and put files on sdcard, follow reboot procedure. All settings will be retained, except:

• root-password. And going forward that password will be disabled everytime after a fw update.
• contents of the rootpartition (tinkerers use that only)

Both issues are already documented for users that use root access here.

Also note that if customers do this manual update before installing the latest version they risk:

• losing gui language
• losing mk2 settings (last status of Multi on/off switch and its current limit setpoint)
• losing ip address settings & wifi password

Offline updating

Besides having the devices update themselves by downloading images from the internet, they must also be updateable from an sdcard and preferably also a usbstick.

Process will be that user needs to put .swu file on sdcard or usbstick. And then start the update from the gui. The main reason for not doing this with the old recovery-image mechanism we know from the ccgx is that there are no accessible buttons on the BeagleBone Controller. Which would be necessary to boot into the sdcard.

Entry in gui will be a separate submenu, below Online updates. It calls the check-updates.sh with a dedicated parameter, to make it look for venus-swu-$machine}.swu or venus-swu-$machine-$build.swu on the root folder of all connected sdcards and usbsticks. It will use the first one it finds for the update.

If it finds nothing, make a speciale failure code. And gui shows that to the user.

Interaction between swupdate and gui

/tmp/swupdate-status

A file showing the result of the last action when swupdate is not running, or the current status in case wupdate is running. Note that the file does not exist until swupdate has been started once. And all swupdate executions must be done with the wrapper scripts, which includes check-updates.sh.

Line 1 contains one number:

-1 = error during checking
-2 = error during downloading and/or installing
 0 = Idle (last check successful -> look at second line if there is a newer version)
 1 = Checking
 2 = Downloading and installing
 3 = Rebooting

Line 2 contains the available firmware version. And is empty or not there when there is no newer version available.

/var/run/versions

First version contains currently running version. Second line contains (if available) version available on other rootfs. File is created after boot by scan-versions.sh

Build output

There are several venus- related images:

• venus-install-sdcard: Full and fresh install, typically used in the factory. Put it contents on a bootable disk, and boot the device with it. It will clean-out whatever is on the emmc, repartition it, and installs the contents of venus.swu on it. Note that 'boot the device with it' is easier said than done on a CCGX!
• venus-swu: the .swu file containing just the rootfs, which can be downloaded/manually installed on the device itself with swupdate.

Useful for CCGX only:

• venus-upgrade-image: put it contents on a bootable disk and boot a CCGX while holding the left key of the circular pad. U-boot on the ccgx-flash will then execute contents on the sd-card: upgrade the ccgx from opkg-based install to swupdate. It will migrate the contents of the existing data partition to the new layout. It is probably also no problem to run it on a device that is already upgraded to swupdate: in that case it simply updates the kernel and rootfs to the latest version, while leaving the /data volume untouched.

And then there is:

• venus-image: the rootfs. Normally not used in itself, use venus-swu instead.

Future work after release

• Once beaglebone is also done, we want to make venus available to raspberrypis. So we'll need swupdate there as well. There is already someone working (Izak) on adding the rpi2 as a machine into the Jethro branches. See here.

• Have a download progress indication

• Implement downloading only deltas: some installations are on an expensive internet connection: only transferring deltas is a nice to have. Requires extra diskspace, which goes hand in hand with more robust resume transfer when downloading. For the CCGX it is perfectly fine to require the user to keep an sdcard inserted for this. Same for the bbb by the way, though that might not be necessary since it already has a ~1.78GB disk builtin.