Conducting TLS Performance Testing - crt26/PQC-LEO GitHub Wiki

Once the testing environment has been properly configured, it is now possible to begin the automated PQC TLS performance testing. This guide provides information on how to execute the testing tool, configure the necessary parameters, and interpret the results. It includes instructions for both single-machine and two-machine testing setups, as well as details on the output format and where to find parsed performance data.

To start the automated testing tool, open a terminal in the scripts/test_scripts directory and run the following command:

./pqc_tls_performance_test.sh

Upon executing the script, the testing tool will prompt you to enter the parameters for the test. Different setup techniques and options will be required depending on the testing scenario (single-machine/two-machine).

It is also recommended to refer to the Testing Options section below before beginning testing to ensure all configurations are correct.

The testing tool will prompt you to enter the parameters for the test. These parameters include:

• Machine type (server or client)
• Whether TLS handshake performance results should be stored (if the machine is a client and energy usage testing is enabled)
• Whether the results should have a custom Machine-ID assigned to them (if the machine is a client and performance results are being stored)
• Duration of each TLS handshake tests (if the machine is a client) †
• Duration of TLS speed tests (if the machine is a client and energy usage testing is not enabled) ††
• Number of test runs to be performed (must match on both machines)
• IP address of the other machine (use 127.0.0.1 for single-machine testing)

† Defines the duration (in seconds) the OpenSSL s_time tool will use for each handshake test window. The client will attempt as many TLS handshakes as possible for each algorithm combination during this period.

Note: Using durations below 5 seconds may produce inf result values for some algorithm combinations. It is recommended to use higher test durations for more consistent results. See the Inf Result Value Occurrence Details section of this page for more information.

†† Defines the duration (in seconds) for benchmarking individual cryptographic operations (for example, signing, key encapsulation, or key derivation) using the OpenSSL speed tool.

If running the full test locally (single-machine), perform the following steps after generating the required certificates:

Server Setup:

1. Run the pqc_tls_performance_test.sh script

2. Select server when prompted

3. Enter the requested test parameters

4. Use 127.0.0.1 as the IP address for the other machine

5. Once the server setup is complete, leave the terminal open and proceed to the client setup

Client Setup:

1. In a separate terminal session, run the pqc_tls_performance_test.sh script again.

2. Select client when prompted

3. Enter the requested test parameters

4. Use 127.0.0.1 as the IP address for the other machine

5. The test will begin, and results will be stored and processed automatically

When two machines are used for testing that are connected over a physical/virtual network, one machine will be configured as the server and the other as the client. Before starting, please ensure that both machines have the same server certificates and private keys stored in the test_data/keys directory.

Server Machine Setup:

1. On the server machine, run the pqc_tls_performance_test.sh script

2. Select the server and enter the test parameters when prompted

3. Use the IP address of the client machine when prompted

4. Now, begin the setup of the client machine before the testing can begin

Client Setup:

1. On the client machine, run the pqc_tls_performance_test.sh script

2. Select the client and enter the test parameters

3. Use the IP address of the server machine

4. Begin testing and allow the script to complete

After standard TLS performance testing completes, raw performance results are saved to the following directory:

test_data/up_results/tls_performance/machine_x

Where machine_x refers to the assigned Machine-ID. If no ID was specified, the default ID of 1 is used.

By default, the TLS performance testing script automatically triggers the parsing process. It passes the Machine-ID and number of test runs to the parsing tool, which processes the raw output into structured CSV files.

These parsed results are saved in:

test_data/results/tls_performance/machine_x

TLS speed results are separated into pqc, hybrid, and classic subdirectories under each machine's speed_results directory. Classical signature results are further separated into RSA and EC/Ed CSV files, while classical key-exchange results are separated into ECDH and XDH CSV files. Per-run CSVs and averages across all configured runs are generated for each category.

Note: When using multiple machines for testing, the results will only be stored on the client machine, not the server machine.

When energy usage testing is enabled, these client-side performance results are not stored by default. The client configuration includes a prompt that allows the user to retain them if required. If retained, only TLS handshake results are stored and parsed because TLS speed testing is not performed in this mode. Energy usage results are stored and parsed separately on the collection machine.

To skip automatic parsing and only output the raw test results, pass the --disable-result-parsing flag when launching the test script:

./pqc_tls_performance_test.sh --disable-result-parsing

For complete details on parsing functionality and a breakdown of the collected PQC TLS performance metrics, refer to the following documentation:

• Parsing Performance Results Usage Guide
• Performance Metrics Guide

Certain PQC, Hybrid-PQC, and classical algorithm combinations may produce inf values for the "Connections Per User Second" metric during TLS handshake testing when using shorter test durations (typically below 5 seconds).

PQC-LEO includes safeguards to:

• warn users when a selected test duration may produce inf values
• exclude rows containing inf from average calculations while tracking the number of valid runs used

For each algorithm combination, session ID first-use and reuse results are handled independently. If no valid runs remain, N/A is used for the averaged metrics and the number of runs used is recorded as 0.

It is recommended to use TLS handshake test durations of 5 seconds or greater to minimise the likelihood of inf values occurring. However, this does not guarantee that all runs can be used in average calculations.

For a detailed explanation of this behaviour and how it is handled, please refer to:

• TLS Handshake Inf Result Handling