Automated Testing Tools - crt26/pqc-evaluation-tools GitHub Wiki
Included Automated Testing Tools
This page provides a high-level overview of the automated benchmarking tools included in the PQC-Evaluation-Tools project.
The repository provides two categories of automated PQC benchmarking:
-
Computational Performance Testing – Benchmarks the standalone performance of PQC cryptographic operations, gathering data on CPU and memory usage.
-
TLS Performance Testing – Benchmarks PQC, Hybrid-PQC, and classic algorithms integrated into the TLS 1.3 protocol, including handshake and cryptographic operation performance.
The testing tools are located in the scripts/test_scripts directory and are fully automated. The tools support assigning custom machine-IDs to the gathered results to make it easy to compare performance on differing systems.
Notice: The HQC KEM algorithms are disabled by default in recent versions of both Liboqs and the OQS-Provider, due to their current implementations not conforming to the latest specification, which includes important security fixes. For benchmarking purposes, the setup process includes an optional flag to enable HQC in these libraries, accompanied by a user confirmation prompt and warning. Enabling HQC is done at the user's own discretion, and this project assumes no responsibility for its use. For instructions on enabling HQC, see the Advanced Setup Configuration Guide, and refer to the Project Dependencies for more information on this issue.
Computational Performance Testing
This tool benchmarks CPU and memory usage for various PQC algorithms supported by the Liboqs library. It produces detailed performance metrics for each tested algorithm.
For detailed usage instructions, please refer to:
PQC Computational Performance Testing
Notice: Memory profiling for Falcon algorithm variants is currently non-functional on ARM systems due to issues with the scheme and the Valgrind Massif tool. Please see the bug report for details. Testing and parsing remain fully functional for all other algorithms.
TLS Performance Testing
This tool benchmarks the performance of PQC, Hybrid-PQC, and classical algorithms when used in the TLS 1.3 protocol. It utilises the PQC implementations natively available in OpenSSL 3.5.0 and those added via the OQS-Provider.
It conducts two types of testing:
-
TLS handshake performance testing – Measures the performance of PQC and Hybrid-PQC algorithms during TLS 1.3 handshakes.
-
Cryptographic operation benchmarking – Measures the CPU performance of individual PQC/Hybrid-PQC digital signature and Key Encapsulation Mechanism (KEM) cryptographic operations when integrated within OpenSSL.
Testing can be performed on a single machine or across two machines connected via a physical/virtual network. While the multi-machine setup involves additional configuration, it is fully supported by the automation tools.
For detailed usage instructions, please refer to:
Testing Output Files
After the testing has been completed, unparsed results and automatically parsed results will be stored in the generated test_data/ directory:
Unparsed Results: test_data/up_results/
Parsed Results: test_data/results/