Testing environment - ligurio/openbsd-tests GitHub Wiki

What should trigger a testrun?

• per-patch tests: patchwork (Setup patchwork on OpenBSD)
• per-snapshot tests: snapmon.py
• by schedule: via crontab(1) or Continuous Integration system (for example Concourse CI)

Where to run the tests?

OpenBSD supports many hardware platforms. There are two types of environment: physical hardware and virtual instance. There are some pros at contras for each of them. Definitely physical hardware is better because it is an environment used by the most part of OpenBSD users and with physical hardware you avoid bugs and side effects brought to you by virtualization. But not everyone are ready to buy and have machine room for testing purpose. So using virtualization is a tradeoff here.

Virtual instance in a cloud

There are many cloud providers which allows to use your own images: Amazon EC2, Google Cloud Engine, Digital Ocean, Azure and providers using CloudStack (for example Exoscale, I use it personally). Regular testing requires rebuilding image with OpenBSD for each new snapshot.

Building of OpenBSD virtual appliance is possible with:

• Packer: VirtualBox, VMware, QEMU and others
• cloud-openbsd script: Amazon AWS and Microsoft Azure

Physical hardware

Using physical hardware requires some resources from you: power energy, free space. You should have infrastructure to allow install and upgrade OpenBSD in unattended mode on your hardware. Learn autoinstall(8) for details. Using serial console connected to system under test is highly recommended.

Hardware-as-a-Service - see https://github.com/testingress/wiki/wiki/Remote-Access-To-Hardware

Tuning OpenBSD before testing

It is recommended to tune OpenBSD no matter where you will run it. Steps below helps you to investigate problems in case of bugs.

• IPMI in hardware may help you to get remote access to the display and keyboard when operating system is unaccessible. Learn manpage ipmi(4) to know how to setup it.
• hardware watchdog help machines automatically recover from system hangs. Watchdog is useful when IPMI device is not available in your hardware. Learn watchdogd(8) for the details.
• steps below helps to gather required information before sending bug report.
  • sysctl ddb.console=1
  • sysctl ddb.panic=1
  • sysctl ddb.log=1
  • ulimit -u

or automatically using Ansible playbook.