Conducting TLS Performance Testing - crt26/pqc-evaluation-tools GitHub Wiki

Conducting PQC TLS Performance Testing

Once the testing environment has been properly configured, it is now possible to begin the automated PQC TLS performance testing. This guide provides information on how to execute the testing tool, configure the necessary parameters, and interpret the results. It includes instructions for both single-machine and two-machine testing setups, as well as details on the output format and where to find parsed performance data.

Testing Tool Execution

To start the automated testing tool, open a terminal in the scripts/testing-scripts directory and run the following command:

./full-oqs-provider-test.sh

Upon executing the script, the testing tool will prompt you to enter the parameters for the test. Different setup techniques and options will be required depending on the testing scenario (single-machine/two-machine).

It is also recommended to refer to the Testing Options section below before beginning testing to ensure all configurations are correct.

Testing Options

The testing tool will prompt you to enter the parameters for the test. These parameters include:

  • Machine type (server or client)
  • Machine Comparison Option
  • Machine Results ID (if comparison option selected)
  • Number of test runs to be performed
  • Duration of each TLS handshake tests (if machine is client) †
  • Duration of TLS speed tests (if machine is client) ††
  • IP address of the other machine (use 127.0.0.1 for single-machine testing)

† Defines the duration (in seconds) the OpenSSL s_time tool will use for each handshake test window. The client will attempt as many TLS handshakes as possible for each algorithm combination during this period.

†† Defines the duration (in seconds) for benchmarking individual cryptographic operations (e.g., signing or key encapsulation) using the OpenSSL s_speed tool.

Single Machine Testing

If running the full test locally (single-machine), perform the following steps after generating the required certificates:

Server Setup:

  1. Run the full-oqs-provider-test.sh script

  2. Select server when prompted

  3. Enter the requested test parameters

  4. Use 127.0.0.1 as the IP address for the other machine

  5. Once the server setup is complete, leave the terminal open and proceed to the client setup

Client Setup:

  1. In a separate terminal session, run the full-oqs-provider-test.sh script again.

  2. Select client when prompted

  3. Enter the requested test parameters

  4. Use 127.0.0.1 as the IP address for the other machine

  5. The test will begin, and results will be stored automatically

Separate Server and Client Machine Testing

When two machines are used for testing that are connected over a physical/virtual network, one machine will be configured as the server and the other as the client. Before starting, please ensure that both machines have the same server certificates and private keys stored in the test-data/keys directory.

Server Machine Setup:

  1. On the server machine, run the full-oqs-provider-test.sh script

  2. Select the server and enter the test parameters when prompted

  3. Use the IP address of the client machine when prompted

  4. Now begin the setup of the client machine before the testing can begin

Client Setup:

  1. On the client machine, run the full-oqs-provider-test.sh script

  2. Select the client and enter the test parameters

  3. Use the IP address of the server machine

  4. Begin testing and allow the script to complete

Outputted Performance Results

After the testing has completed, all the unparsed results will be stored in the test-data/up-results/ops-provider/machine-x directory. This directory contains the TLS handshake performance and cryptographic speed test results for PQC, Hybrid-PQC, and classic ciphersuites. Results are organised by the machine-ID assigned during the testing setup.

These raw output files are not yet ready for interpretation or graph generation. To parse the data into a format that can be used for further analysis, please refer to the Parsing Results section in the main README file.

For a detailed description of the OQS-Provider TLS performance metrics that this project can gather, what they mean, and how this project scripts structure the un-parsed and parsed data, please refer to the Performance Metrics Guide.

Note: When using multiple machines for testing, the results will only be stored on the client machine, not the server machine.