Magma's releases are split between two categories: major and minor releases. Major releases (Ex. 1.2.0, 1.3.0) include planned features and are accompanied by an in-depth test report. Minor releases (Ex. 1.3.1, 1.3.2) are primarily for bug fixes and minor content updates. Some “new” features/content may slip in if not ready at time major release was cut.

The release notes and name can be found on the Github Releases page and include test reports, key features and improvements, known issues, bug fixes, logs, and source code.

The Magma release process is community driven and based on the needs of contributors. Community members are encouraged to create a release channel for each upcoming release where the community can align on what features to be worked on for the upcoming version. Each release should be led by one organization, ideally the group that has the largest code contribution for the upcoming release. The organization will be responsible for planning the features for the release, the timelines, and communication structure to the broader Magma community. Prior to a release candidate being cut, the community must first ensure that master is healthy. From there no additional code will be added and the candidate release will go through three main phases:

• Development - code creation, pull requests, CI checks, etc.
• Validation - system integration testing, performance testing, stability testing, etc.
• Release - the release is tagged, build created, communication to community, etc.

If the candidate passes all of the steps above then it will be an official release. If issues are identified as part of this process then they will be fixed on master in order to restore it to a healthy status. One that is complete, a new release candidate can be cut.

Releases should be conducted on a quarterly or shorter basis in order to minimize the amount of testing effort required. Minor releases can be conducted over a period of weeks.

1. Release Planning Template

• Used to aggregate features under development across the Magma community and to align on the subset of features that will be included in the subsequent release. Additionally, the timeline tab can be used to share with the broader community the important milestone dates around cutting the release candidate and the conclusion of testing.

2. Release Tracker

• Used to track the status for the set of features selected for the release. This provides project management functionality and visibility to the Magma community to track deliverables and understand the status of the release. Additionally, issues can be tracked separately in the second tab. This document can be reviewed by the community during the weekly release meeting.

3. Release Reports

• Used to share with the Magma community the features and changes made for the release. This report includes test reports, install scripts/logs, critical bug fixes, and key features/improvement descriptions.

⚠️ This documentation is loosely based on the 1.8.0 release. After 1.8.0 the Magma artifactory was switched. This included a more restrictive publishing of artifacts - only CI workflows are allowed to publish. That is, some parts might need to be adapted or extended for the next releases or in general should be automated for future releases. Please update this documentation accordingly.

⚠️ This documentation assumes that the AGW is built with Bazel.

At some point the community has decided that a release is planned. Usually the master needs to be brought into a release state, e.g., some features need to be finished, bugs need fixing, etc. Now the release can start.

It is recommended that a dedicated release team is set up with dedicated contact persons that are required to approve merges on the release branch and for decisions on the general procedure and on special circumstances. This team should also organize communication to the community and end users, including release notes and announcements. These steps are not in scope of this documentation.

The general idea is to create a release branch, publish all relevant artifacts to the artifactory test repositories, test the artifacts and eventually promote them to production repositories.

• Apply a merge freeze on master - this can be done by an announcement in Slack. There is a GitHub app "mergefreeze" that should be able to enforce the merge freeze (by someone with sufficient rights on the magma GH repo).
• Create a local branch with the name v1.x, where x is the new minor version to be released.
• Push the branch.
• The merge freeze can now already be lifted.
• ⚠️ All builds that are triggered with this push are still based on artifacts in the focal-ci distribution of the magma-packages-test repository. There will already be Docker images with the tag 1.x created. These images can be ignored - they will be replaced in follow-up steps. Also, failing workflows should be ignored for now.

First, the third party artifacts need to be created and pushed to magma-packages-test/focal-1.x.0. This can be done by manually running

• "Magma Build & Publish 3rd Party Dependencies" (.github/workflows/magma-build-3rd-party.yml)
  • with distribution focal-1.x.0
• "Magma Build & Publish Open vSwitch Artifacts" (.github/workflows/magma-build-openvswitch.yml)
  • with distribution focal-1.x.0
• ⚠️ Note that only people in @magma/approvers-ci can run these workflows manually.

Verify that the workflows are successful and that artifacts have been created in https://linuxfoundation.jfrog.io/ui/native/magma-packages-test/pool/focal-1.x.0/.

At this point it might be that something failed and needs a fix. You should distinguish between changes that are only needed on the release branch and changes that are needed on master and the release branch. If a change is only relevant for the release branch then create a PR versus this branch. If a change is also needed on master then follow the backporting documentation.

The following changes should be done in one PR (but best in multiple commits). ⚠️ This list might be incomplete and should be extended if something is discovered to be missing.

In various places Docker images used in workflows and Dockerfiles must be pinned (from latest) to the current tag when the release branch was created. The sha tag of the respective images can be found on the Magma GitHub packages overview.

• "AGW Build, Format & Test Bazel" (.github/workflows/bazel.yml)
  • Pin the images to the latest build by changing the latest tag to the respective sha. Best search for :latest and change the tag accordingly.
  • Change all occurences of focal-ci to focal-1.x.0.
  • ⚠️ TODO: workflow needs to be enabled for runs on pull requests on release branches!
• "AGW Build, Publish & Test Container" (.github/workflows/agw-build-publish-container.yml)
  • ⚠️ TODO: workflow currently does not push with release tag (1.x) if run on a release branch. This needs to be fixed then this entry can be removed as there should be no adaption necessary on this workflow.
• "Magma Build & Publish" (.github/workflows/build_all.yml)
  • There should be no change necessary here. Docker images should already be published with the 1.x tag. Debian artifacts should be published via "AGW Build, Format & Test Bazel".
• "AGW Generate Coverage" (.github/workflows/agw-coverage.yml)
  • Pin the image of BAZEL_CACHE_PLAIN_IMAGE.
• "AGW Lint" (.github/workflows/agw-workflow.yml)
  • Pin the image of BAZEL_CACHE_PLAIN_IMAGE.

• cwf/gateway/docker/c/Dockerfile
  • Pin the ghcr.io/magma/magma/bazel-cache-plain image.

In general, in almost all Dockerfiles, change focal-ci to focal-1.x.0. Currently, the only Dockerfile where this is not needed is .devcontainer/bazel-base/Dockerfile. You can just do a full text search for focal-ci in Dockerfiles and replace the entry. E.g., in lte/gateway/docker/services/python/Dockerfile.

Workflows using a VM (e.g., integration tests) should also use a VM setup based on the release test branch. In orc8r/tools/ansible/roles/pkgrepo/tasks/main.yml change focal-ci to focal-1.x.0.

Bump the Helm chart versions for the release. You can use https://github.com/magma/magma/pull/13731 as a guide.

After the changes above are done and pushed, the CI should be monitored for any failures. All failures need to be analyzed and fixed if necessary.

Verify that the following artifacts have been pushed to the artifactory:

• magma, magma-sctpd and magma-dhcp-cli to https://linuxfoundation.jfrog.io/ui/native/magma-packages-test/pool/focal-1.x.0/
• All Docker images to magma-docker-<COMPONENT>-test/<IMAGE> with tag 1.x on https://linuxfoundation.jfrog.io/ui/repos/tree/General
• All Helm charts to https://linuxfoundation.jfrog.io/ui/repos/tree/General/magma-helm-test with file name <COMPONENT>-1.x.0.tgz

In general all workflows should be successful on a release branch. That is, all runs should be monitored and analyzed. Especially the integration tests can be very flaky and might need multiple restarts.

In general new features should be covered by automated tests, but it also makes sense to verify them manually. The community is responsible for testing all relevant workflows related to installing and upgrading the different components of Magma, doing load tests, etc.

After all tests are successful it is time to do the actual release.

⚠️ Open question: should the Dockerfiles be based on magma-packages-test or magma-packages-prod?

Tag the current head of the release branch v1.x as v1.x.0. Note that the release branch can receive new commits, but the tag must not be moved. The tag marks the revision on which the released artifacts are based.

Manually run the following workflows to promote the Magma artifacts from the test repositories to the prod repositories.

• "Magma Promote Debian Packages" (.github/workflows/debian-packages-promote.yml)
  • with distribution focal-1.x.0
  • ⚠️ TODO only copy last versions
• "Magma Promote Docker Images" (.github/workflows/docker-promote.yml)
  • with branch_tag 1.x
  • with release_tag 1.x
• "Magma Promote Helm Charts" (.github/workflows/helm-promote.yml)
  • with magma_version 1.x.0

• Cut the docs for the 1.x release on the master branch, see https://github.com/magma/magma/wiki/Contributing-Documentation#versioning-process
• Publish new development artifacs on master with version 1.(x+1).0, i.e., if 1.9.0 was just released, then artifacts on master that contain a version should now be build for 1.10.0.
  • in lte/gateway/release/BUILD.bazel change the VERSION variable

It should only be necessary to create a hot fix release (1.x.1) if the released artifacts need to be fixed. This would require the complete process documented here but using 1.x.1 (1.x.2, etc).

If release artifacts are not affected, then it is sufficient to make changes on the release branch directly or backport changes.

• Wrong or damaged artifacs were published.
  • Coordinate deletion of these artifacts with the LF support via #questions-to-linux-foundation-it.
  • If this happens too often it might make sense to create a manual workflow with sufficient arguments that can take care of cleanups.

It is strongly recommended that the release process becomes more streamlined.

• More parts should be modeled by CI workflows (e.g., load testing, HiL testing, ...).
• Existing workflows should be pipelined.
• Complexity should be removed.
• Existing tests should be more stabilized.

... a goal could even be a one-click-release.