NOTE: This document it is not static and is likely to change and evolve over time.

See Release Process for general release information.

This test plan outlines the (mostly) manual testing that is done prior to releases. Future release should take the work collected here and automate it. For CI/automated testing, see automated testing.

The following systems are for now, out of scope for this testing:

• Secure boot
• Container runtime and guests

Test output spreadsheet available HERE

Board Support Packages (BSPs) are provided for a number of other switches outlined in the mion supported switches. Several of the switches on the page are marked as reference switches, meaning that they are the main hardware platforms used for development, testing and validation by the project.

Release testing is intended to find any new bugs or regressions which might not have otherwise been noticed during the development cycle so they can be fixed.

The current expectation is that these tests will be run manually with future releases moving what can be automated into the automated QA testing. The resulting bugs/regressions will need to be either fixed (if relatively trivial) or logged and triaged for future releases, if necessary.

Before logging any bugs/regressions, make sure that there a issue doesn't already exist in any of the GitHub repos. FIXME add a link to the query here

• If it is clear where in the code the issue is, log it directly to the correct GitHub repo and add the issue to the "TODO" column of the mion project board

• If you're not clear where the issue belongs, use the bug/feature template and it will triaged by the team

All new issues will have to be triaged prior to the release process and either closed, fixed or added to the errata, as required.

---

Test that the system builds when starting from an empty directory and the any instructions in the docs are correct.

These tests are intended to check that the main documentation is correct and can be used to create a valid image which can be flashed to a switch and will boot up.

• [C000] Build an image using a clean checkout of mion (sub repos and submodules) with an empty sstate and build directory
  • Verify that the build completes without errors
  • Verify that a valid ONIE image is created
• [C001] Walk through the steps outlined in the Getting Started Guide
  • Verify that the steps outlined int the document are correct and make sense
  • Verify that the build completes without errors
  • Verify that a valid ONIE image is created

Using one of the reference switches:

• [C010] Boot the test system (reference switch) into ONIE* and install the image using onie-nos-install <image_url> or ``onie-nos-install <local_file>`
  • Verify that the install completes sucessfully and reboots
• [C011] Verify that the GRUB/u-boot menu entry is correct and boots

* NOTE: It is expected that the ONIE version installed is relatively recent

• TODO Add a test for ONIE tools when they have been fixed
• TODO check that we can read the ONIE version info

These tests are intended to check that the system boots correctly and that the OS itself is working correctly with no new or unexpected issues:

• [C100] System boots with no error messages in the log (dmesg)
• [C101] Check for failed systemd services using systemctl systemctl list-units --failed
• [C102] Power cycle the switch and confirm that there are no unexpected errors
• [C103] Ensure package manager is working with opkg update
• [C104] Build image with ptest support with ptest-runner installed.
• [C105] Run ptest-runner, directing the output to <VENDOR><MACHINE>.log and commit the file to mion-testing, saving to <RELASE NAME>_ptests/. If older log exists, replace. Review file for any failed tests. If time permits, add SUMMARY: at top of log with a list of failed tests.

• [P001] Check for ONLP errors in the boot sequence with dmesg
• [P002] Run onlps and verify that the output is correct (per machine)

• [P???] Run onlps chassis onie show
• [P???] Run onlps chassis asset show
• [P???] Run onlps chassis env
• [P???] Run onlps sfp inventory
• [P???] Run onlps sfp bitmaps
• [P???] Run onlps sfp dev read <port> <devaddr> <addr> <len> eg onlps sfp dev read 49 0x50 0 256
  • FIXME This test needs to be defined per system and presumably requires hardware to be plugged in
• [P???] Run onlps platform manager run 30
• [P???] Run onlps fan percentage get 1; sleep 5; onlps fan percentage set 1 100 You will want to reset the fan speed with the got percentage in the first part of the command
• [P???] Run onlps oid to json psu-1 debug-all
• [P???] Run onlps chassis debug show

DENT testing should be fairly similar to ONLP testing.

TODO

This section is about exercising and testing the various ASIC SDEs

TODO