Release Checklist - betrusted-io/betrusted-wiki GitHub Wiki

This is the release checklist (when creating a new tagged minor revision):

Note: the EC release build environment is on ci.betrusted.io and relies on the toolchain at the following path:

RISCV_TOOLS=/tools/xpack-riscv-none-elf-gcc-12.2.0-3
FOMU_TOOLS=/opt/fomu-toolchain-updated
export PATH=$RISCV_TOOLS/bin:$FOMU_TOOLS/bin:$PATH

Prepping the release

  • Roughly 0.5 hrs (+1.0 hrs if SoC has an update)
  • No warnings in hosted, renode, or hardware builds
  • Trim buildable apps (currently mtxchat is in CI but should not be in release)
  • All CI tests passing
  • Update tag on EC, rebuild, tag & push (on ci.betrusted.io in bunnie/code/betrusted-ec, see note about build env above)
    • git tag -a "v0.9.x" -m "commit message" Tag to release version
    • betrusted_ec.py --revision=pvt to generate an FPGA image with the tag in it
    • cargo xtask copy-precursors to absorb the FPGA image
    • git commit -a to commit the updated FPGA image to the tree
    • do not delete the current tag and re-tag -- the parser expects a +n (in this case +1) at the end, a clean tag omits that and version reporting breaks.
    • cargo xtask hw-image to test the final composition
    • git push --tags to send tags to the cloud
    • git push origin to update the actual repo
    • Check that CI triggered. It may not trigger if it's just a tag update and no code changes.
    • Mark build as "keep build forever" for both betrusted-ec and betrusted-ec-fw
  • Update tag on SoC, rebuild, tag & push
    • git tag -a "v0.9.x" -m "commit message" Tag to release version
    • .\betrusted_soc.py -e .\dummy.nky Generate a fresh SoC image that contains the tag
    • git push --tags to send the new tag to the cloud
    • git push origin to push the actual repo
    • Check that CI triggered. It may not trigger if it's just a tag update and no code changes.
    • Check slack histograms on release build for any suspicious timing closures (max and min)
    • Check the candidate SoC file across 3 hardware test cases: nom, corner-0.95 and corner-1.0 (requires swapping out the mainboard)
    • Mark the build artifacts as "keep forever" for betrusted-soc
  • Absorb SoC artifact into xous-core; rebuild, tag & push
    • cd precursors && ./update_soc_pvt.sh - absorb the production SoC image into the Xous tree. Note: this does not absorb any SVD changes, this is assumed to have been done previously to the release process!
    • Push latest SoC ref to crates.io:
      • edit utralib to reflect the change in git rev
      • use publish.py to bump version numbers & publish new revs
      • edit verifier.rs/main.rs in xtask to absorb latest version numbers
    • cargo xtask app-image - confirm everything builds
    • git commit -a to commit the updated SoC precursors to the tree
    • git tag -a "v0.9.x" -m "commit message" Tag to release version
    • git push --tags then git push to send new tag to the cloud and kick off the final CI

Testing the release

Roughly 2-3 hours if all goes well. Note that it may make sense to push the release prior to all testing being done, because the backup/restore tests rely on finding the release kernel in the release location.

  • Simple proof-of-run in hosted mode
    • cargo xtask run vault
  • Simple proof-of-run with Japanese and Chinese locale in hosted mode
    • cargo xtask run vault --feature locales/lang-[ja,zh]
  • Simple proof-of-run in renode
    • cargo xtask renode-image.
    • renode emulation/xous-release.resc
  • Update from previous version to current version using production update mechanism: python3 -m precursorupdater -b
  • Functionality tests not covered by CI:
    • Check audio (now done with OQC test)
    • Check vibe (now done with OQC test)
    • Test net server, net tcpget, time, and ping
    • Run suspend/resume loop for 50 cycles minimum without failure
  • Simple proof-of-run with text-to-speech enabled on hardware
    • Rely on the TTS image built on the CI server
    • python3 -m precursorupdater -b -l en-tts - load the TTS image onto the hardware
    • Note: if TTS fails consider doing cargo update in espeak-embedded, and re-running the espeak-embedded build task to refresh the APIs on the external server artifact
  • Updating tests: (note: important to run on next EC update after v0.9.15 as changes were made to the EC update loop in v0.9.15 - remove this note once we have passed that point)
    • Provision unit with previous "stable" image, and update successfully to the new tag.
      • Now:
        • Use factory-newify repo and run rollback.sh script to roll a device back to the previous release without erasing the PDDB
        • Force a gateware update to move the SoC to the older version without erasing keys
        • Confirm all artifacts (soc, xous, ec, wf200) are at the previous version.
        • Perform an update using python3 -m precursorupdater -b and confirm all artifacts are at the correct version.
      • Previously (for historical records):
        • Use factory-newify repo and run the newify.sh script to roll a device back to 0.9.8 release.
        • Run update_ci -s and confirm that the auto-update flow succeeds (note: this has to be done after the "Latest" release has been updated on the server, so it sort of blends in with the first step of "pushing the release")
  • Run the OQC factory test
    • Make sure the test peer AP is booted and active
    • Run test oqc on device that is provisioned with the candidate release firmware.
  • Confirm that the factory-test jenkins CI flow has no errors this without erasing PDDB)
  • Backup/restore test:
    • Take a device that is running the current tag + initialized keys + PDDB.
    • If the BIP-39 flow was modified in this release, provision BBRAM keys with bbram_helper.py.
    • Create a secret basis and store something in it.
    • Prepare and run a backup: prepare backup from menu and then run tools/backup.py
    • Newify the device.
    • Restore the backup by running tools/restore.py
    • Confirm the secret basis data is present
    • Check backup integrity: python3 ./backalyzer.py -c -f <path to backup>
    • Note: this test flow does not test the following: If BBRAM key step is skipped, then BIP-39 key entry of BBRAM/eFuse key is not tested; rekeying to a different device is not tested.

Pushing the Release

  • About 0.5hrs
  • Make the release official:
    • Copy the release artifacts on the server:
      • mkdir /var/releases/v0.9.x
      • cp /var/latest-ci/* var/releases/v0.9.x
      • rm /var/releases/latest - remove old symlink
      • cd /var/releases && ln -s v0.9.x latest - create new symlink
    • For Baochip, see below
    • Update the CI expected version:
      • edit, on ci-pi, the version.expect script to have the new version inside the ci branch on bootstrap-mainboard repo
    • Update restore.py to include the latest release in URL_LIST
    • Update and tag the release on the factory tester:
      • In the factory test environment, check out main branch and change to bootstrap-mainboard/precursors/ and run update_ci.sh
      • This is done so that any potential regressions/changes since the test of the factory test are captured (ideally nothing has changed, though)
      • Commit the artifacts, tag & push. Revert to ci branch.
      • (best practice) newify, reboot & test one last time using the START button
      • Call the factory and ask them to absorb the changes.
    • Snapshot a factory-newify repository -- so we can quickly reproduce an RMA that may be specific to a quirk of the repository state.
      • create a new branch for the relevant version in factory-newify
      • inside the precursors directory run fetch_binaries.sh v0.9.x. This will snag the latest binaries from the named release directory and also update the VERSION file.
      • commit and push the branch. Do not merge to main. main is the first version shipped, version 0.9.5
    • Capture the crate audit
      • These commands are run inside the crate_scraper repo
      • cp ../xous-core/Cargo.lock . - update the .lock file to the latest version
      • rm ./crates/* - clear out the old crates fetched
      • ./crate_scraper.py --download - fetch the crates
      • rm -r ./builds/* && rm builds.rs and then ./crate_scraper.py --analyze - create the build.rs audit. The "Not found in archive" errors are normal.
      • Update the README.md with the xous-core tag used in the scrape and other notes
      • Commit & push
      • There is a potential TOCTOU because the crates are scraped sometime after the release build is actually built (within hours of the build, but still, a window for a timing attack).

Baochip Addendum

  • About 1-2 hours(?) may be shorter as I get better at it
  • Sign the artifacts using bao-signing and the command ./wrapper.sh --sig bao1 --name bao1-feb17 --toolchain 441559d7e1984623ec0a52f60f90240c740b6c41 where --sig is the signing key used, --name is a free-form name that generally takes the pattern of - and --toolchain is the hash of the baobit toolchain used to build the release (see https://github.com/sbellem/baobit)
  • This will result in the release being copied to the staging area, currently bao-ci:code/releases/signed/
  • Test the signed results by triggering the bao1x-signed-reserialize job on CI
  • Place the artifacts on the server. For each relevant directory:
    • Delete the artifact directory created by the previous set of commands - as these are unsigned, CI-generated artifacts
    • Replace the artifact directory with the signed elements (do not overwrite - all old artifacts must be removed, as some name patterns don't match).
    • "Relevant directory" means an artifact that is intended to be released. A whole subset of images are created for testing, but not all of them may be appropriate for release. As of the last update of these instructions, only the bootloader is updated. However, not all releases get a new bootloader! The bootloader tracks the chip lot, not the firmware train.
Add a custom footer
⚠️ **GitHub.com Fallback** ⚠️