Creating, Testing, and Publishing Magma VM Base Images - magma/magma GitHub Wiki
The VMs used in Magma are handled with Vagrant. During usage, they are built on top of pre-built base images that are currently hosted in the Vagrant Cloud. This documentation describes how these base images can be created, tested, and published.
⚠️ The Magmacore Artifactory must support this operating system/version!
As a first step, prepare a manual test with the new OS version by changing the corresponding vm.box entry in lte/gateway/Vagrantfile to the generic target one (e.g. ubuntu/focal64) and adapting the box version to the one stated in the corresponding Vagrant Cloud account. Find the Ansible playbook of the VM and set preburn = true and full_provision = false. Boot the VM with these settings.
HOST [magma/lte/gateway]$ vagrant up <name>`
After the initial boot, shut down the VM via vagrant halt <name>, revert the changes to the flags in the Ansible playbook, and fully provision it.
HOST [magma/lte/gateway]$ vagrant up <name> --provision && vagrant ssh <name>
Fix possible issues (missing packages, changed dependencies, deprecations, ...). Delete/adapt steps specifically aimed at the previous OS.
For feature changes, start again with preburn = true and full_provision = false. You can now edit the Ansible playbooks to make the changes you want for the VM. The two-step provisioning is the same as described above. Test all features/possible use-cases for the VM.
The packaging process is done via Packer. When updating the VM OS: adapt boot_command, iso_url, iso_checksum, etc. for the new OS in the corresponding JSON in orc8r/tools/packer (some templates can be found here). All Ansible roles (even if their jobs are full_provision-only) must be included in the "role_paths" JSON entry.
HOST [magma/orc8r/tools/packer]$ packer build <JSON-file>.json
This packs the VM at its default path magma/orc8r/tools/packer/builds/<name>.box.
For Vagrant to know about the base image, it has to be registered first.
HOST [magma/lte/gateway]$ vagrant box add path/to/box.box --name <name> --force
Check the list of VMs with vagrant box list and adapt the respective vm.box = <name> and vm.box_version = <version> in the lte/gateway/Vagrantfile with this information (by default, vm.box_version = 0 should do it). You may have to remove the vm.box_version constraint to get it to run a local box, otherwise there will be a Ruby error.
The VM can now be booted. Perform all the respective unit and/or integration tests in the new environment.
- AGW Unit Tests
- AGW Sudo Python Tests
- S1AP LTE Integration Tests (also with a containerized AGW)
- S1AP Federated Integration Tests
- CWAG Integration Tests
- Connect Orchestrator and build NMS following the Quick Start Guide
The Linux Foundation can provide the corresponding token for the upload of the base image to the Magmacore account in the Vagrant Cloud. The upload is done via a script which extracts the VM's name and provider from the filename and the version number from the current date.
HOST [magma/orc8r/tools/packer]$ export VAGRANT_CLOUD_TOKEN=<token>
HOST [magma/orc8r/tools/packer]$ ./vagrant-box-upload.sh path/to/box.box
In general, uploaded boxes must be published by a developer holding admin rights to the Magmacore Vagrant Cloud account. This step is necessary for CI testing. The vagrant-box-upload.sh script publishes boxes automatically. A (buggy) uploaded box can be deleted via the Vagrant Cloud CLI, for example, vagrant cloud version delete magmacore/magma_dev $VERSION.
In order to download the new build by default, a PR containing the new version of the uploaded box in the lte/gateway/Vagrantfile is needed. The versions are timestamped to the upload date, e.g.1.2.20221012.
The VMs are cached in GitHub to avoid rate limits with the Vagrant Cloud. The cache keys for the updated VMs have to be updated (new version) for them to be re-cached. Full-text search the repository for ~/vagrant.d/boxes for occurrences (the cache in the CWF integ tests contains the trfserver) . For consistency, these caches are named after the VM versions.
After the changes in the Ansible files, Vagrant files, and box-caches have been merged to master, the VMs need to be re-built locally. This means destroying the VMs built with the old images and provisioning them again with vagrant up. By doing so, the new base images are downloaded and all updated Ansible roles will be applied to the images. This ensures an up-to-date system for development and testing. For example, for the AGW VMs, one can do the following after pulling the master branch:
HOST [magma/lte/gateway]$ vagrant destroy # destroys all AGW VMs
HOST [magma/lte/gateway]$ vagrant up # build the VMs with new base images
Note that a draft for a CI workflow to build, publish, and release a virtual machine can be found in PR #14777.