Feature: System Updates for dnf, yum, apt hosts - cockpit-project/cockpit GitHub Wiki

Summary

Cockpit provides a "Software Updates" page for Atomic/OSTree. Extend it for "classic" yum/dnf/apt based operating systems: Show available changes, provide button for applying outstanding updates, and allow the user to enable automatic updates.

Stories

User stories:

Roy administrates his company's mail server. He normally applies updates once a week, but today he reads on IT news about an urgent Postfix security issue.

Moss administers his team's internal file server. He wants to cut down on the amount of time he spends on manually doing software updates on the server.

Workflows:

Roy logs in to Cockpit and sees that there is a Postfix update available referring to that CVE, so he sends out an announcement about a short planned downtime, the applies the update and reboots the server. He verifies that the mail server comes back online.

Some days later on, Roy does the weekly routine update on the server. After applying the update is complete, the page shows a notification that the machine needs to be restarted for the changes to take effect (as they included a kernel update). He does that, grabs a coffee, and when he returns, verifies that the machine came back up alright.

Moss logs in to the server via Cockpit. He enables automatic installation of updates. Every few months some update causes connection errors as a reboot is required, then Moss does that manually.

Scope

Operating systems supported by Cockpit have some standard mechanism to manually or automatically apply updates and describe the changes in available updates. They usually allow some configuration, such as whether to only download or to actually apply updates automatically, whether to apply them during runtime or shutdown, and the interval at which this happens. Cockpit should just provide a simple interface to this existing mechanism and not try and invent any logic of its own. It should only support the most common use cases and its UI and complexity should be comparable to the existing "Software Updates" page for OSTree.

Phase 1 provides showing the availability of updates, describing them to the extent possible, and provide a button to apply them, together with a live logtail of the update output. This can be quite complex and packages often show custom error messages, so Cockpit should not try and interpret them, but just show the output verbatim.

Phase 2 provides a simple check box to enable daily automatic updates, (to be discussed, check if all backends support this) possibly with a choice between applying them during runtime (default) or at shutdown.

Cockpit's currently supported operating systems use one out of four package managers for which this needs to be implemented: dnf (Fedora), yum (Centos/RHEL 7), apt (Debian/Ubuntu), and ostree (Atomic).

OS interfaces: Feature summary

The subsequent sections contain the details how to use/configure these features.

Feature	yum	dnf	apt	PackageKit	ostree
Detailled changelogs	✔	✔	?	(✔)	✗
Install only security updates	✔	✔	(✔)	✔	✗
Reboot required notifications	✗	(✔)	Ubuntu only	✗ (broken)	always
Automatic updates	✔	✔	✔	✗	(✔)
... also weekly	✗	✔	✔	✗	(✔)
... security only	✔	✔	✔	✗	✗

OS interfaces: View changes and manual application

dnf:

Human-readable and structured description of available updates can be seen with

# dnf updateinfo info
===============================================================================
kernel-4.10.13-200.fc25
===============================================================================
Update ID : FEDORA-2017-17d1c05236
Type : security
Updated : 2017-05-01 14:13:41
Bugs : 1445207 - CVE-2017-7477 kernel: net: Heap overflow in skb_to_sgvec in macsec.c
: 1446566 - CVE-2016-9604 kernel: security: The built-in keyrings for security tokens can be joined as a session and then modified by the root user [fedora-all]
: 1445208 - CVE-2017-7477 kernel: net: Heap overflow in skb_to_sgvec in macsec.c [fedora-all]
: 1389433 - CVE-2016-9604 kernel: security: The built-in keyrings for security tokens can be joined as a session and then modified by the root user
Description : The 4.10.13 stable update contains a number of important fixes across the tree.

This should at least be broken into individual packages (list with expanders for details) and get line wrapped. This can be refined to recognize bug and CVE references and turn them into proper links.

Updates are applied with dnf -y update.

Showing only security updates: dnf updateinfo info security. There is no direct command to only install security updates, but the package list can be built from the former command and then installed by name.

There is no mechanism for packages to flag that a reboot is required for the changes to take effect. However, if Tracer is installed (python3-dnf-plugins-extras-tracer package in Fedora), this can be called after the update to see if services, the session, or the whole system need to be restarted.

yum:

yum-plugin-changelog only displays the raw RPM spec changelogs, and only as part of yum update. So this needs to be run with --assumeno to not actually apply the update, and the changelog sections need to be parsed out of the other output:

# yum install -y yum-plugin-changelog
# yum --changelog update
Loaded plugins: auto-update-debuginfo, changelog, fastestmirror, product-id,
              : search-disabled-repos, subscription-manager
[...]
Resolving Dependencies
--> Running transaction check
---> Package NetworkManager.x86_64 1:1.4.0-17.el7_3 will be updated
[...]
--> Finished Dependency Resolution

Changes in packages about to be updated:

ChangeLog for: 1:NetworkManager-1.4.0-19.el7_3.x86_64,
             : 1:NetworkManager-libnm-1.4.0-19.el7_3.x86_64,
             : 1:NetworkManager-team-1.4.0-19.el7_3.x86_64,
             : 1:NetworkManager-tui-1.4.0-19.el7_3.x86_64
* Wed Mar 29 08:00:00 2017 Beniamino Galvani <[email protected]> - 1:1.4.0-19
- core: alyways force a sync of the default route (rh#1431268)

* Mon Mar 27 08:00:00 2017 Beniamino Galvani <[email protected]> - 1:1.4.0-18
- manager: ensure proper disposal of unrealized devices (rh#1433303)
[...]
Dependencies Resolved

================================================================================
 Package                  Arch   Version                   Repository      Size
================================================================================
Installing:
 kernel                   x86_64 3.10.0-514.16.1.el7       updates         37 M
[...]

Updates are applied with yum -y update.

yum updateinfo list security all is supposed to list only security updates, and `yum -y update --security' limits installation to only security updates (see documentation). However, this does not seem to work on CentOS 7, all updates are of type "normal" there. This might just be due to missing annotations in CentOS?

There is no mechanism for packages to flag that a reboot is required for the changes to take effect.

apt:

The standard mechanism is apt-listchanges, but this is currently tightly integrated with apt-get dist-upgrade, i. e. it displays the changes after downloading the updates and before applying them. There is currently no good way to only display changes without actually installing them ("dry run"). For that reason, Ubuntu's update-manager has its own logic for downloading the changelogs from http://changelogs.ubuntu.com/ and displaying the relevant part. But this package is not available in Debian. PackageKit's changelog retrieval seems to work, though.

TODO: research some better solution here, if we need this at all.

As a first step, apt-get -qq update; apt list --upgradable will at least provide the number, names, and versions of packages for which updates are available.

Updates are applied with apt-get -y dist-upgrade.

There is no convenient way to only install security updates, as that's usually chosen by the admin via enabling or disabling the apt sources for -updates (i. e. non-security bug fixes). It can be done through some hacks though, if necessary.

Ubuntu packages create a /run/reboot-required flag on upgrade if a reboot is necessary for the changes to take effect. Some Debian packages inherit that, but the most critical ones (particularly the kernel) don't.

PackageKit:

PackageKit's API has an API for determining available updates and getting details/changelogs for them. E. g. on CentOS 7 or Fedora 25:

# pkcon get-updates
Available   	tzdata-2017b-1.fc25.noarch (updates)                        	Timezone data

# # pkcon get-update-detail tzdata
Resolving                     [=========================]         More than one package matches:
1. tzdata-2017b-1.fc25.noarch [updates]
2. tzdata-2016h-1.fc25.noarch [fedora]
Please choose the correct package: 1
                              [=========================]         
Getting update details        [=========================]         
Starting                      [=========================]         
Finished                      [=========================]         
Details about the update:
 Package: tzdata-2017b-1.fc25.noarch
 Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1434206
 Update text:     Resolves: #1434206
    - Rebase to tzdata-2017b
    - Haiti began observing DST on March 12, 2017.
 Changes: 
 State: stable
 Issued: 
 Updated:

The required interaction is just an artifact/bug of pkcon, when using the D-Bus API this is not an issue.

This does not currently work on Ubuntu 16.04 LTS, there the PackageKit daemon crashes with a segfault in libpk_backend_aptcc.so. It does work on Ubuntu 17.04 and Debian testing. On Debian 8 it does not crash, but does not deliver any information.

PackageKit's GetUpdates mentions classifying updates for security vs. bug fix, but this is broken for dnf and not useful on CentOS/Debian stable either as that does not seem to classify package updates accordingly (see yum paragraph above).

PackageKit's UpdateDetail has a restart field, but it is not implemented in dnf and yum (just for apt), so it cannot be reliably used for restart required notifications on Fedora/CentOS/RHEL. There is also a RequireRestart signal which is implemented by (only) the apt backend. So we should make use of those interfaces on apt based systems at least.

The main issue is that PackageKit is often not installed in server environments and is not currently a dependency of Cockpit.

ostree:

Already implemented, to the degree possible (there are no individual package changelogs):

Screenshot of Software Updates on Atomic

Atomic is designed that the actual application of a new image is through a reboot, so a reboot notification should be shown whenever the running image is older than the current one.

OS interfaces: Automatic application

Reference for dnf/yum: https://fedoraproject.org/wiki/AutoUpdates

dnf:

Enable automatic download of updates:

dnf install dnf-automatic
systemctl enable --now dnf-automatic-install.timer

Defaults to daily; weekly is supported by creating a dnf-automatic.timer.d drop-in that overrides OnUnitInactiveSec=7d. This can also be used to run the update at a more specific time.

Defaults to applying all updates; can be restricted to security updates only with upgrade_type = security in /etc/dnf/automatic.conf.

Has no builtin support for automatic reboot. This can be done with a systemd service unit drop-in that runs the reboot in an ExecStartPost= command.

yum:

Enable automatic download of updates:

yum install -y yum-cron
systemctl enable yum-cron && systemctl start yum-cron

(The service unit only creates the flag file that gates the cron job.)

Enable automatic application of updates by setting apply_updates = yes in etc/yum/yum-cron.conf. Parse the value to get the current mode.

Only mode of operation is daily, as this uses /etc/cron.daily/. No unintrusive way to change.

Defaults to applying all updates; can be restricted to security updates only with update_cmd = security in etc/yum/yum-cron.conf.

There is no support or hook for doing automatic reboots after applying updates.

apt:

Enable automatic download of updates:

apt-get install -y unattended-upgrades

Enable automatic application of updates and rebooting when necessary:

printf 'APT::Periodic::Unattended-Upgrade 7;\nUnattended-Upgrade::Automatic-Reboot-WithUsers "true";\n' > /etc/apt/apt.conf.d/51-enable-unattended-upgrades

(See defaults in /etc/apt/apt.conf.d/50unattended-upgrades.)

Check current status with:

apt-config shell UnattendedUpgradeInterval APT::Periodic::Unattended-Upgrade

Empty or UnattendedUpgradeInterval='0': not enabled; when running the above command it will say UnattendedUpgradeInterval='7'

Defaults to daily; weekly is supported by changing UnattendedUpgradeInterval as above. More specific times can be set via drop-ins of apt-daily-upgrade.timer.

Defaults to applying all updates; can be restricted to security updates only by setting the apt option Unattended-Upgrade::Origins-Pattern { "label=Debian-Security" }; for Debian, or Unattended-Upgrade::Allowed-Origins { "${distro_id} ${distro_codename}-security"; } for Ubuntu. (TODO: The latter might work on Debian too, to be investigated).

PackageKit:

PackageKit has no API for controlling automatic upgrades.

ostree:

There is no documented official way in the documentation (projectatomic or wiki). Also, Atomic is meant to be rebooted after installing a new image. To be discussed: It may be too intrusive to reboot unannounced in the background. If an admin wants this functionality, Cockpit could install a cron job that does atomic host upgrade and shutdown -r +5m if a newer image got downloaded.

Prior art

GNOME Software - Updates
GNOME Software - Installed packages
Ubuntu update-manager
Wordpress - Updates Guide Screenshot
Windows Server 1 2
Synology Diskstation 5.0 Article Video
MacOS App store screenshot1, screenshot2

UI Design

Requirements:

Checking for updates and reading the changes takes some 10s up to ~ 2 minutes; so this probably requires a "Check for updates..." button and some spinner.
Make it possible to only install security updates when manually applying.
If a "reboot required" notification is available, show that on the Software Updates page after application of th updates is complete.
If we do a daily vs. weekly selection, hide it if that feature is not available (i. e. on RHEL/Centos with yum).

Mockup

Updates overview

Installation progresss

Mockup Notes

List

Security list and package list have been merged.
Security issues are at the top of the list and highlighted (here, with a red background — subject to change)
Packages are sorted alphabetically in the list within groups (security and non-security updates)
Bug #s are optional, but should be shown and linked to the appropriate bug page when available.

Security Updates

When a package update is marked as a security fix, an icon (currently fa-bug in the mockup) and a "Security Update" label (in red & semi-bold weight) is the top line of the details section.
If CVEs are included in the package changelog, they are appended after "Security Update" (which gets a colon :) and link to their appropriate CVE info page
If a CVE includes a description, it shall be included in a tooltip when hovering over the CVE #
If there is no description for a CVE, hovering the link should show "(No description currently available.)"
The package changelog should be shown below the security update line

Appendix

Prototype implementation is on a separate GitHub project as it's much faster to iterate that way.
Implementation for proper Cockpit inclusion in PR #6724
It might be useful to show the package manager logs, in order to help with investigating regressions and behaviour changess. This is orthogonal to this specification though.