Release Checklist - betrusted-io/betrusted-wiki GitHub Wiki

This is the release checklist (when creating a new tagged minor revision):

Note: the EC release build environment is on ci.betrusted.io and relies on the toolchain at the following path:

RISCV_TOOLS=/tools/xpack-riscv-none-elf-gcc-12.2.0-3
FOMU_TOOLS=/opt/fomu-toolchain-updated
export PATH=$RISCV_TOOLS/bin:$FOMU_TOOLS/bin:$PATH

Prepping the release

  • Roughly 0.5 hrs (+1.0 hrs if SoC has an update)
  • No warnings in hosted, renode, or hardware builds
  • Trim buildable apps (currently mtxchat is in CI but should not be in release)
  • All CI tests passing
  • Update tag on EC, rebuild, tag & push (on ci.betrusted.io in bunnie/code/betrusted-ec, see note about build env above)
    • git tag -a "v0.9.x" -m "commit message" Tag to release version
    • betrusted_ec.py --revision=pvt to generate an FPGA image with the tag in it
    • cargo xtask copy-precursors to absorb the FPGA image
    • git commit -a to commit the updated FPGA image to the tree
    • do not delete the current tag and re-tag -- the parser expects a +n (in this case +1) at the end, a clean tag omits that and version reporting breaks.
    • cargo xtask hw-image to test the final composition
    • git push --tags to send tags to the cloud
    • git push origin to update the actual repo
    • Check that CI triggered. It may not trigger if it's just a tag update and no code changes.
    • Mark build as "keep build forever" for both betrusted-ec and betrusted-ec-fw
  • Update tag on SoC, rebuild, tag & push
    • git tag -a "v0.9.x" -m "commit message" Tag to release version
    • .\betrusted_soc.py -e .\dummy.nky Generate a fresh SoC image that contains the tag
    • git push --tags to send the new tag to the cloud
    • git push origin to push the actual repo
    • Check that CI triggered. It may not trigger if it's just a tag update and no code changes.
    • Check slack histograms on release build for any suspicious timing closures (max and min)
    • Check the candidate SoC file across 3 hardware test cases: nom, corner-0.95 and corner-1.0 (requires swapping out the mainboard)
    • Mark the build artifacts as "keep forever" for betrusted-soc
  • Absorb SoC artifact into xous-core; rebuild, tag & push
    • cd precursors && ./update_soc_pvt.sh - absorb the production SoC image into the Xous tree. Note: this does not absorb any SVD changes, this is assumed to have been done previously to the release process!
    • Push latest SoC ref to crates.io:
      • edit utralib to reflect the change in git rev
      • use publish.py to bump version numbers & publish new revs
      • edit verifier.rs/main.rs in xtask to absorb latest version numbers
    • cargo xtask app-image - confirm everything builds
    • git commit -a to commit the updated SoC precursors to the tree
    • git tag -a "v0.9.x" -m "commit message" Tag to release version
    • git push --tags then git push to send new tag to the cloud and kick off the final CI

Testing the release

Roughly 2-3 hours if all goes well. Note that it may make sense to push the release prior to all testing being done, because the backup/restore tests rely on finding the release kernel in the release location.

  • Simple proof-of-run in hosted mode
    • cargo xtask run vault
  • Simple proof-of-run with Japanese and Chinese locale in hosted mode
    • cargo xtask run vault --feature locales/lang-[ja,zh]
  • Simple proof-of-run in renode
    • cargo xtask renode-image.
    • renode emulation/xous-release.resc
  • Update from previous version to current version using production update mechanism: python3 -m precursorupdater -b
  • Functionality tests not covered by CI:
    • Check audio (now done with OQC test)
    • Check vibe (now done with OQC test)
    • Test net server, net tcpget, time, and ping
    • Run suspend/resume loop for 50 cycles minimum without failure
  • Simple proof-of-run with text-to-speech enabled on hardware
    • Rely on the TTS image built on the CI server
    • python3 -m precursorupdater -b -l en-tts - load the TTS image onto the hardware
    • Note: if TTS fails consider doing cargo update in espeak-embedded, and re-running the espeak-embedded build task to refresh the APIs on the external server artifact
  • Updating tests: (note: important to run on next EC update after v0.9.15 as changes were made to the EC update loop in v0.9.15 - remove this note once we have passed that point)
    • Provision unit with previous "stable" image, and update successfully to the new tag.
      • Now:
        • Use factory-newify repo and run rollback.sh script to roll a device back to the previous release without erasing the PDDB
        • Force a gateware update to move the SoC to the older version without erasing keys
        • Confirm all artifacts (soc, xous, ec, wf200) are at the previous version.
        • Perform an update using python3 -m precursorupdater -b and confirm all artifacts are at the correct version.
      • Previously (for historical records):
        • Use factory-newify repo and run the newify.sh script to roll a device back to 0.9.8 release.
        • Run update_ci -s and confirm that the auto-update flow succeeds (note: this has to be done after the "Latest" release has been updated on the server, so it sort of blends in with the first step of "pushing the release")
  • Run the OQC factory test
    • Make sure the test peer AP is booted and active
    • Run test oqc on device that is provisioned with the candidate release firmware.
  • Confirm that the factory-test jenkins CI flow has no errors this without erasing PDDB)
  • Backup/restore test:
    • Take a device that is running the current tag + initialized keys + PDDB.
    • If the BIP-39 flow was modified in this release, provision BBRAM keys with bbram_helper.py.
    • Create a secret basis and store something in it.
    • Prepare and run a backup: prepare backup from menu and then run tools/backup.py
    • Newify the device.
    • Restore the backup by running tools/restore.py
    • Confirm the secret basis data is present
    • Check backup integrity: python3 ./backalyzer.py -c -f <path to backup>
    • Note: this test flow does not test the following: If BBRAM key step is skipped, then BIP-39 key entry of BBRAM/eFuse key is not tested; rekeying to a different device is not tested.

Pushing the Release

  • About 0.5hrs
  • Make the release official:
    • Copy the release artifacts on the server:
      • mkdir /var/releases/v0.9.x
      • cp /var/latest-ci/* var/releases/v0.9.x
      • rm /var/releases/latest - remove old symlink
      • cd /var/releases && ln -s v0.9.x latest - create new symlink
    • Update the CI expected version:
      • edit, on ci-pi, the version.expect script to have the new version inside the ci branch on bootstrap-mainboard repo
    • Update restore.py to include the latest release in URL_LIST
    • Update and tag the release on the factory tester:
      • In the factory test environment, check out main branch and change to bootstrap-mainboard/precursors/ and run update_ci.sh
      • This is done so that any potential regressions/changes since the test of the factory test are captured (ideally nothing has changed, though)
      • Commit the artifacts, tag & push. Revert to ci branch.
      • (best practice) newify, reboot & test one last time using the START button
      • Call the factory and ask them to absorb the changes.
    • Snapshot a factory-newify repository -- so we can quickly reproduce an RMA that may be specific to a quirk of the repository state.
      • create a new branch for the relevant version in factory-newify
      • inside the precursors directory run fetch_binaries.sh v0.9.x. This will snag the latest binaries from the named release directory and also update the VERSION file.
      • commit and push the branch. Do not merge to main. main is the first version shipped, version 0.9.5
    • Capture the crate audit
      • These commands are run inside the crate_scraper repo
      • cp ../xous-core/Cargo.lock . - update the .lock file to the latest version
      • rm ./crates/* - clear out the old crates fetched
      • ./crate_scraper.py --download - fetch the crates
      • rm -r ./builds/* && rm builds.rs and then ./crate_scraper.py --analyze - create the build.rs audit. The "Not found in archive" errors are normal.
      • Update the README.md with the xous-core tag used in the scrape and other notes
      • Commit & push
      • There is a potential TOCTOU because the crates are scraped sometime after the release build is actually built (within hours of the build, but still, a window for a timing attack).