Automated Testing Tools - crt26/PQC-LEO GitHub Wiki

This page provides a high-level overview of the automated benchmarking tools included in the PQC-LEO project.

The repository provides four categories of automated PQC benchmarking:

• Computational Performance Testing – Benchmarks the standalone performance of PQC cryptographic operations, gathering data on CPU and peak memory usage.

• TLS Performance Testing – Benchmarks PQC, Hybrid-PQC, and classical TLS 1.3 handshake configurations and cryptographic operation performance.

• TLS Handshake Transmission Cost Testing – Measures the bytes sent and received during one-way and mutually authenticated TLS 1.3 handshakes.

• PQC Energy Usage Testing – Benchmarks the energy consumption of PQC algorithms during computational and TLS performance testing using supported energy meters.

The testing tools are located in the scripts/test_scripts directory and are fully automated. The tools support assigning custom machine-IDs to the gathered results to make it easy to compare performance on differing systems.

This tool benchmarks CPU and peak memory usage for various PQC algorithms supported by the Liboqs library. It produces detailed performance metrics for each tested algorithm.

For detailed usage instructions, please refer to:

PQC Computational Performance Testing

Notice: Memory profiling for Falcon algorithm variants is currently non-functional on ARM systems due to issues with the scheme and the Valgrind Massif tool. Please see the bug report for details. Testing and parsing remain fully functional for all other algorithms.

This tool benchmarks the performance of PQC, Hybrid-PQC, and classical algorithms when used in the TLS 1.3 protocol. It utilises the PQC implementations natively available in OpenSSL 4.0.3 and those added via the OQS-Provider.

It conducts two types of testing:

• TLS handshake performance testing – Measures the performance of PQC, Hybrid-PQC, and classical algorithms during TLS 1.3 handshakes.

• Cryptographic operation benchmarking – Measures the CPU performance of individual PQC/Hybrid-PQC digital signature and Key Encapsulation Mechanism (KEM) operations, together with classical digital signature and key-exchange operations, when integrated within OpenSSL.

Testing can be performed on a single machine or across two machines connected via a physical/virtual network. While the multi-machine setup involves additional configuration, it is fully supported by the automation tools.

For detailed usage instructions, please refer to:

PQC TLS Performance Testing

Notice: Certain PQC, Hybrid-PQC, and classical algorithm combinations may produce values of inf for "Connections Per User Second" results in TLS handshake testing when using smaller testing windows. Please refer to the TLS Handshake Inf Result Handling page for further information.

This tool measures the amount of data exchanged during TLS 1.3 handshakes using PQC, Hybrid-PQC, and classical algorithm configurations. It tests every supported signing algorithm and KEM pairing for PQC and Hybrid-PQC, and every supported signing algorithm, key exchange group, and ciphersuite combination for classical TLS. Each configuration is tested using both one-way and mutual authentication and the bytes sent, bytes received, and total bytes during the handshake are recorded from the client's perspective.

The test runs locally over the loopback interface and writes final CSV results directly to the results directory.

For detailed usage instructions, please refer to:

TLS Handshake Transmission Cost Testing Instructions

This tool benchmarks the energy consumption of supported cryptographic algorithms during computational and TLS testing, including PQC, Hybrid-PQC, and classical configurations where applicable. It provides tools and automation scripts to collect energy usage metrics using supported energy meters and a collection device. It supports automation for the following categories of energy usage testing:

• Computational Performance Energy Testing
• TLS Handshake Energy Testing
• TLS Operations Energy Testing

This component of the PQC-LEO framework includes two core elements, the automated energy usage testing scripts and the energy collector tools. The energy collector tools provide a flexible way in which to orchestrate testing between collection and testing devices using serial or network communications and provide a modular way to integrate support for different energy meters. The automated energy usage testing scripts provide an easy to use interface for utilising the energy collector tools.

Current support for energy meter devices is limited, but the framework is designed to allow the integration of additional meters in the future in a simple as possible way.

For detailed usage instructions, please refer to:

Energy Usage Testing

Notice: Certain OpenSSL Hybrid-PQC KEM algorithms that are supported for normal TLS speed performance testing are not supported for TLS operations energy usage testing due to limitations in encoder support. Please refer to OpenSSL algorithms section of the Supported Algorithms page for further details.

By default, performance and energy test results are automatically parsed at the end of their testing scripts. The TLS handshake transmission cost tool instead writes structured CSV files directly and does not use the automated parsing process.

Parsed results will be stored in the following directories, depending on which testing was performed:

• test_data/results/computational_performance/machine_x
• test_data/results/tls_performance/machine_x
• test_data/results/tls_handshake_bytes/machine_x
• test_data/results/energy_test_results/[test_type]/machine_x

machine_x is the Machine-ID number assigned to the results when executing the testing scripts. If no custom Machine-ID is assigned, the default ID of 1 will be used.

For energy usage testing results, test_type is pqc_performance_energy_results, tls_handshake_energy_results, or tls_operations_energy_results, depending on the type of results parsed.

If needed, automatic parsing can be disabled when calling the testing scripts by passing a flag to the testing script. This then facilitates the manual calling of the Python parsing scripts.

For complete details on parsing functionality and a breakdown of the collected performance metrics, refer to the following documentation:

• Performance Metrics Guide
• Parsing Performance Results Usage Guide