Automated Testing Tools - crt26/pqc-evaluation-tools GitHub Wiki

Included Automated Testing Tools

This page provides a high-level overview of the automated benchmarking tools included in the project which are located in the scripts/test-scripts directory.

The project provides two categories of automated benchmarking:

  • Liboqs Performance Testing - Used for gathering PQC computational performance data

  • OQS-Provider TLS Performance Testing - Used for gathering PQC TLS 1.3 networking performance benchmarking when integrated into OpenSSL 3.4.1

The testing tools are located in the scripts/test-scripts directory and are fully automated. The tools support multi-machine testing, with the option to assign a machine ID when executing the testing scripts.

Liboqs Performance Testing

This tool benchmarks CPU and memory usage for various PQC algorithms supported by the Liboqs library. It produces detailed performance metrics for each tested algorithm.

For detailed usage instructions, please refer to following wiki page:

PQC Computational Performance Testing

Note: HQC KEM algorithms are disabled by default in the latest Liboqs version due to a known vulnerability. The main setup script provides an option to enable HQC for benchmarking if required. Please refer to the Advanced Setup Configuration Guide page for further details.

OQS-Provider TLS Performance Testing

This tool is focused on benchmarking the performance of PQC and Hybrid-PQC algorithms when integrated within OpenSSL (3.4.1) via the OQS-Provider library.

It conducts two types of testing:

  • TLS handshake performance testing – Measures the performance of PQC and Hybrid-PQC algorithms during TLS 1.3 handshakes.

  • Cryptographic operation benchmarking – Measures the CPU performance of individual PQC/Hybrid-PQC digital signature and Key Encapsulation Mechanism (KEM) cryptographic operations when integrated within OpenSSL.

Testing can be performed on a single machine or across two machines connected via a physical/virtual network. While the multi-machine setup involves additional configuration, it is fully supported by the automation tools.

For detailed usage instructions, please refer to following wiki page:

PQC TLS Performance Testing

Note: The following signature algorithms are excluded from the automated TLS benchmarking due to known incompatibilities with RFC 8446:

  • UOV-based schemes (e.g., OV_Is, OV_III, and their hybrid variants)
  • CROSSrsdp256small

These algorithms remain available for computational benchmarking using the Liboqs tools.

Testing Output Files

After the testing has been completed, unparsed results will be stored in the test-data/up-results directory:

  • Liboqs results: test-data/up-results/liboqs/machine-x/

  • OQS-Provider results: test-data/up-results/oqs-provider/machine-x/

Where machine-x refers to the machine ID assigned at the beginning of the test. This ID is used to organise output when running tests across multiple machines.