Testing - notro/gud GitHub Wiki

GUD repo tests

There are some tests in the repo directory tests.

Dependencies

Python >= 3.6 is needed because of the F-strings.

This is how I install the dependencies on Raspberry Pi OS:

$ sudo apt install python3-pytest
$ sudo pip3 install pyusb lz4 numpy

libdrm

$ sudo apt -y install meson libtool xutils-dev

$ git clone https://gitlab.freedesktop.org/mesa/drm.git libdrm
$ cd libdrm

$ meson build -Dinstall-test-programs=true
$ ninja -C build
$ sudo ninja -C build install

pykms

$ sudo apt-get install -y cmake libfmt-dev

$ git clone https://github.com/tomba/kmsxx
$ cd kmsxx/
$ git submodule update --init

$ # If the display uses the new USB connector type:
$ # (libdrm doesn't have DRM_MODE_CONNECTOR_USB yet, so it's hardcoded)
$ curl https://gist.githubusercontent.com/notro/1379ba97e1cbdea847d02688b75d2805/raw/0966305fcbf1b0e794b9de86caf84b1ff43bedbd/connector-Add-new-DRM-connector-types.patch | git am

$ meson build
$ ninja -C build
$ sudo ninja -C build install

$ sudo ldconfig

$ # Put pykms in sys.path (python3 -m site):
$ echo /usr/local/lib/arm-linux-gnueabihf/python3.7/site-packages | sudo tee /usr/local/lib/python3.7/dist-packages/arm-linux-gnueabihf.pth

To avoid running tests as root, add these rules:

/etc/udev/rules.d/70-gud-test.rules

ACTION=="add", SUBSYSTEM=="usb", ATTR{idVendor}=="1d50", ATTR{idProduct}=="614d", TAG+="uaccess"
ACTION=="add", SUBSYSTEM=="usb", ATTR{idVendor}=="16d0", ATTR{idProduct}=="10a9", TAG+="uaccess"

ACTION=="add", SUBSYSTEM=="backlight", KERNEL=="card*", RUN+="/bin/chmod 666 /sys/class/backlight/%k/brightness"

pytest

pytest is used to run the tests/test_*.py files.

tests/test_direct.py contains tests that unbinds the kernel driver and talks directly to the device over USB.

# Run all tests
$ pytest-3

# Show test names
$ pytest-3 -v

# Exit on first failure
$ pytest-3 -x

# Don't run those 'direct' tests that locks up some devices some of the time
$ pytest-3 -m 'not stall'

# Don't run the direct tests at all
$ pytest-3 -m 'not direct'

# Don't run the grid test
$ pytest-3 -k 'not test_grid'

# Show why some tests are skipped
$ pytest-3 -rs

# Don't wait (5 secs) between tests
$ pytest-3 --test-delay=0

# Don't wait 5 secs between tests, but wait for the user to press a key
$ pytest-3 -s

# Don't test XRGB8888 since it just an emulation format for this RGB565 device
$ pytest-3 --xrgb8888=RGB565

# Test just one test and wait for keypress before disabling the display
$ pytest-3 -v -s test_modes.py::TestModes::test_smpte[1920x1080-RGB565]

Some useful command line flags:

  -k EXPRESSION         only run tests which match the given substring expression.
  -m MARKEXPR           only run tests matching given mark expression.
  -x, --exitfirst       exit instantly on first error or failed test.
  -s                    Don't capture output.
                        In this test setup it makes the test runner wait for a key press between tests.
  -v, --verbose         increase verbosity. Show test names.
  -r chars              show extra test summary info e.g. -rfEs gives skip reason + the default value.
  --setup-show          show setup of fixtures while executing tests.

custom options:
  --xrgb8888={R1,XRGB1111,RGB565}
                        XRGB8888 emulated format
  --test-delay=TEST_DELAY
                        Delay between tests (default 5 secs)

Other tests

Performance

Tests throughput using various compression ratios:

perf-direct.py: Test USB device directly
perf-kms.py: Test through the host driver (requires the async_flush module parameter)

Compression ratios:

No compression (only perf-direct.py)
x0: Fully random image that will fail to compress into the max buffer size, so will fallback to no compression, so this takes the hit of first trying the compression.
x1: Random image with enough zeroes to compress into the same size as a no compress image.
x2,3,4,8,16: Fill image with zeroes until it compresses to the desired ratio.

# Performance tests using various compression ratios
$ python3 tests/perf-{direct,kms}.py

# Use default mode, RGB565 format, skip compressions and keep display enabled.
$ python3 tests/perf-direct.py -p -f RGB565 -n -k

# Rebind kernel driver after using perf-direct.py
$ python3 -c "import gud; gud.find_first_setup().attach_kernel_driver()"

See wiki/Performance

USB Compliance Tests

The USB Implementers Forum provides tools to verify that a given device adheres to the USB standard.

The USB Command Verifier is only available on Windows machines with a xHCI controller:

USB20CV for USB 2.0
USB3CV for USB 3.2
I didn't find one for USB 1.1

The Chapter 9 Tests should pass for a GUD device.

Links:

Linux usbtest

The Linux kernel has a usbtest module which has some Chapter 9 tests and some other control tests. The tests are run using the testusb tool.

The device needs to be set appropriately as a usb test device.

# Make sure gud isn't bound
$ sudo modprobe -r gud

$ sudo modprobe usbtest vendor=0x1d50 product=0x614d

# plug in device

$ dmesg
[424132.794826] usb 1-1.4: new full-speed USB device number 29 using xhci_hcd
[424133.404733] usb 1-1.4: new high-speed USB device number 30 using xhci_hcd
[424133.540640] usbtest 1-1.4:1.0: matched module params, vend=0x1d50 prod=0x614d
[424133.540692] usbtest 1-1.4:1.0: Generic USB device
[424133.540718] usbtest 1-1.4:1.0: high-speed {control} tests

# find sysfs path
$ lsusb -t
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 5000M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/1p, 480M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 480M
        |__ Port 2: Dev 4, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M
        |__ Port 2: Dev 4, If 1, Class=Human Interface Device, Driver=usbhid, 1.5M
        |__ Port 4: Dev 30, If 0, Class=Vendor Specific Class, Driver=usbtest, 480M

# /dev/bus/usb/BUS/DEV
$ export DEVICE=/dev/bus/usb/001/030

$ sudo -E ~/testusb -t 9 -c 1
/home/pi/testusb: /dev/bus/usb/001/030 may see only control tests
/dev/bus/usb/001/030 test 9,    0.000865 secs

$ sudo -E ~/testusb -t 10 -c 1 -g 15
/home/pi/testusb: /dev/bus/usb/001/030 may see only control tests
/dev/bus/usb/001/030 test 10,    0.003431 secs

$ dmesg
[424153.667506] usbtest 1-1.4:1.0: TEST 9:  ch9 (subset) control tests, 1 times
[424194.610864] usbtest 1-1.4:1.0: TEST 10:  queue 15 control calls, 1 times