Release Process

Our releases are highly automated. Contact Martin, Allison, or ask on Matrix for help if you need to use it or something broke with it.

We release every other Wednesday. Detailed release procedure:

1. Issues and PRs that are urgent for the next release should have a release-blocker label. Review the list, and ask team members if they still have anything for the current release. Check cockpit-machines and cockpit-podman as well.

2. Find out the previous release: PREV=$(git describe --abbrev=0 --tags); echo $PREV

3. Look at the list of closed issues with the release-note label for likely inclusions into the release notes. Review the changes since the previous release with git shortlog $PREV.. and pick out other noteworthy bits. Stick to new features and ignore changes to tests and bug fixes. Do the same for our other projects, in particular cockpit-podman and cockpit-machines.

4. From that, draft a new release note blog post on the cockpit-project.org repo in a pull request (from your own forked repo). Describe user-visible changes from a user's (not developer's) point of view. Include screencasts, screenshots, and thank external contributors. Use the main branch in your repository, then you get an automatically rendered page once you enable GitHub pages for your cloned repo.

   Our web site repository has a generate-release-notes.rb script which automates this process. Use that script to create a first draft, and iterate on changing the original PRs (and re-generating the blog post) for as long as possible to keep them in sync.

   As an example, look at the release notes of Cockpit 169 (with a video) which was done in PR #177, or Cockpit 171 (with external contributions) which was done in PR #185. Both PRs refer to the preview page.

   Let some native English speaker review the blog post for correct spelling, grammar, and legibility. Usually Garrett and Allison. Ask them to start reviewing the headings first (as that blocks the next step), and the contents later.

   Finally, remove the release-note label for the PRs which got included.

5. If you have a GPG key, make sure it's added and verified in your GitHub profile so that the release actually appears as signed. If you don't have nor want one, ask someone else in the team to push the tag for you.

6. The headings of the release notes become the "release news" summary which appears on the GitHub release page and downstream package changelogs. They are taken from the signed release tag.

   Create that tag with: git tag -s $((${PREV%.*} + 1)) (or -a if you don't have a GPG key). As the tag description, use the format as documented in cockpituous release:

   123
   
   - Add this cool new feature
   - Support Python 3
   - Fix wrong color on the bikeshed
   

   Push it to cockpit's main: git push origin $((${PREV%.*} + 1)) (you might use a different origin name).

7. This triggers the GitHub release workflow, which does the upstream GitHub release, updates the guide on the website, and prepares a Cockpit Client Flathub release. It also triggers Packit to do the Fedora and our COPR releases, controlled through packit.yaml.

   The best way to watch the progress and review failure logs is to start on the Cockpit releases page. Click on the tagged commit (7 hex numbers), then click on the Statuses icon to the left of the commit subject (✅, ❌, or 🟡), and click on a "propose-downstream" or "rpm-build" Details link. This gets you to a https://github.com/cockpit-project/cockpit/runs/... page with all the relevant logs and links.

8. Packit will generate Fedora dist-git pull requests for Rawhide and currently supported releases. Wait for them to appear and pass the scratch build and integration test, and merge them. This step is not currently automated, but work is underway to use osbuild fedora-bot. You can use it manually like this:

   git clone https://github.com/osbuild/fedora-bot
   cd fedora-bot
   podman run -it --rm -v .:/bot:ro ghcr.io/osbuild/fedora-bot /bot/fedora_bot.py -c cockpit:2 --apikey YOURPAGUREAPIKEY
   

   You can also run this without an API key for "check mode" only, or generate an API key here.

   Landing the PR will then trigger a corresponding koji build and bodhi update automatically through Packit.

9. Repeat the tagging for the other projects which should be released.

10. Verify that the new version is:

    • current and verified on the GitHub project release page
    • built in COPR
    • current on the cockpit documentation site.
    • present on Fedora bodhi for testing and landing.

11. After COPR is built, run Cockpit's build-ws-container workflow to update the quay.io/cockpit/ws container. (This step needs to be automated)

12. Add the bodhi URLs to the release announcement blog post and merge it.

13. Create a textual release announcement and send it to [email protected] and CC: [email protected] (requires mailing list membership). Also BCC: two Red Hat internal mailing lists (ask Martin about these).

    Use the header, footer, and formatting like in the Cockpit 248 announcement.

14. Ask Martin to upload the new release to Debian. This step is not currently automated.

15. Ask Jelle to upload the new release to Arch Linux. This step is not currently automated.

16. Ask Lis to upload the new release to Flathub. This step is not currently automated.

17. Ask Martin or Garrett to announce the new release on Mastodon

18. Test the packages on bodhi on your machine:

    alias kojiget='koji download-build --arch=noarch --arch=x86_64'
    kojiget cockpit-172-1.fc36
    rpm --freshen --verbose *.rpm
    

    Make sure the update works, go through all the pages in Cockpit (make sure you disable your ~/.local/share/cockpit for this!), and report back on bodhi with a +1. After getting enough karma (usually after a day or two), the package should automatically be pushed to stable.