Skip to content

TestPlan

Phil Burk edited this page Oct 17, 2022 · 5 revisions

Test Plan and QA Strategy for PortAudio

Our goal is to have a set of tests that can be used to quickly verify PortAudio on multiple platforms.

Please see our test status for each platform.

We have several kinds of tests.

  • Self Checking Unit Tests - These do not require a human to verify the results. They return a clear PASS or FAIL.

  • Semi-Automated Loopback tests - These require a human to setup an audio interface with audio loopback cables. The test then outputs audio signals and analyses the recorded result. They return a clear PASS or FAIL.

  • Listening Tests - These require a human to run a program and listen to the results. We are converting these to unit tests.

We recommend that developers periodically run the Self Checking Unit Tests and a few selected Listening Tests before checking in changes to PortAudio.

Self Checking Unit Tests

These tests can be run from the command line. They have a return code of zero if they pass, non-zero if they fail. This allows them to be run as part of an automated test suite.

qa/paqa_errs.c - This test deliberately generates errors then verifies that the appropriate error message is returned.

qa/paqa_devs.c - This test tries to output audio using many combinations of device, numChannels, format, interleave, etc.

qa/paqa_latency.c - This test verifies that the default latencies for each device are non-zero, and that suggestedLatency is honored.

Loopback Test - This test listens to its own output using a loopback cable. It plays sine waves then analyzes the audio input. It can detect pops, noise and phase errors. The loopback tests will ultimately be able to replace most of the Listening Tests. The loopback test can be built on Linux and Mac using "make loopback".

Please see this file for loopback instructions: https://github.com/PortAudio/portaudio/blob/master/qa/loopback/README.txt

Listening Tests

We highly recommend that the following tests and examples be run before checking in changes to portaudio.

  • examples/pa_devs.c - List the available devices and host APIs. Please verify that it seems complete and correct.

  • examples/paex_sine.c - Plays a sine wave using a callback.

  • examples/paex_write_sine.c - Plays a sine wave using a blocking write.

  • examples/paex_record.c - The program will ask you to record a few seconds of audio then play it back. You will need a microphone or other audio source.

  • test/patest_multi_sine.c - Listen for sine wave with no pops or clicks on every channel. This is a good test for N>2 channel devices.

  • test/patest_leftright.c - A tone will be played on the left and right channel. Make sure you can hear both tones.

  • test/patest_stop_playout.c - Tests stopping of a stream in various ways.

  • test/patest_wire.c - Test full duplex input and output on one stream. You should hear the audio input passed to the output.

A list of additional tests can be found here: http://www.portaudio.com/docs/v19-doxydocs-dev/group__test__src.html

Clone this wiki locally