PQC TLS Performance Metrics - crt26/PQC-LEO GitHub Wiki

The TLS performance testing suite benchmarks PQC, Hybrid-PQC, and classical algorithm configurations available through both OpenSSL's native support and the OQS-Provider. OpenSSL 4.0.3 provides native PQC implementations alongside those exposed by the OQS-Provider, and the suite is designed to evaluate performance consistently across the available implementations. It measures performance within the TLS 1.3 handshake protocol and the execution speed of cryptographic operations directly through OpenSSL. This provides insight into how PQC schemes perform in real-world security protocol scenarios. An expanded set of classical digital signature algorithms, key-exchange groups, and ciphersuites is also tested to establish performance baselines for comparison with PQC and Hybrid-PQC algorithms.

As part of the automated TLS testing, two categories of evaluations are conducted:

• TLS Handshake Testing - This simulates full TLS 1.3 handshakes using OpenSSL’s s_server and s_time tools, evaluating both standard and session-resumed connections.

• TLS Speed Testing - This uses the OpenSSL speed tool to benchmark low-level operations such as key generation, encapsulation, signing, verification, and key derivation.

The TLS handshake performance tests measure how efficiently different PQC, Hybrid-PQC, and classical algorithm combinations perform during the TLS 1.3 handshake process. These tests are executed using OpenSSL's built-in benchmarking tools (s_server and s_time).

For PQC and Hybrid-PQC, each test uses a signing-algorithm and KEM pairing. Classical tests instead use a signing-algorithm, key-exchange-group, and ciphersuite combination. Every TLS handshake configuration is performed as many times as possible during the set testing time window, both with and without session ID reuse, to evaluate the overall performance.

The table below describes the performance metrics gathered during this testing:

Metric	Description
Reused Session ID	* marks session reuse; an empty value marks first-use connections.
Connections in User Time	Number of successful TLS connections reported during user CPU time.
User Time (s)	User CPU time consumed by the client during testing.
Connections Per User Second	Number of connections divided by the client's user CPU time in seconds.
Connections in Real Time	Number of successful connections reported alongside elapsed wall clock time.
Real Time (s)	Total wall clock time elapsed, including system I/O and process delays.

When averaged TLS handshake result files are generated, additional columns are included alongside the standard metrics:

Column	Description
Runs used for Average	The number of test runs included in the average calculation.
Total Runs	The total number of test runs performed for the given configuration.

These columns are used to indicate how many test runs were included in each average calculation. This is necessary because, in some cases, a value of inf may be produced for the "Connections per User Second" metric when using shorter TLS handshake test durations.

For PQC, Hybrid-PQC, and classical results, if an inf value occurs in any metric, the entire row is excluded from the average calculation for the affected algorithm combination and session-reuse mode. Where no valid rows remain, the averaged metrics are written as N/A and Runs used for Average is set to 0. Total Runs continues to record the total number of test runs performed. The additional columns record how many rows contributed to each average and are only present for TLS handshake results.

Further information on this behaviour and how it can be avoided is provided in the TLS Handshake Inf Result Handling page.

TLS speed testing benchmarks the raw cryptographic performance of PQC, Hybrid-PQC, and classical algorithms within OpenSSL. It uses both OpenSSL-native algorithms and PQC algorithms provided by the OQS-Provider. The test uses OpenSSL's speed tool to measure operation time and throughput for PQC/Hybrid-PQC KEM and signature algorithms, classical RSA/EC/Ed signature algorithms, and classical ECDH/XDH key-exchange algorithms.

Classical RSA speed results are reported only by modulus size, such as RSA_2048, because openssl speed does not expose a separate RSA-PSS selector or result category. Its standard RSA selectors use the RSA/PKCS#1 v1.5 path. Consequently, standard TLS speed output does not contain RSA-PSS_* rows.

The primary objective of this test is to gather the base system performance of the schemes when integrated into the OpenSSL library. The results provide insight into the algorithm's standalone efficiency when running within OpenSSL, which can produce additional overhead compared to the performance tests provided by the computational performance testing suite.

The following table describes the metrics collected for PQC/Hybrid-PQC signatures and classical RSA signatures during TLS speed testing:

Metric	Description
keygen (s)	Average time in seconds to generate a signature key pair.
sign (s)	Average time in seconds to perform a signing operation.
verify (s)	Average time in seconds to verify a digital signature.
keygens/s	Number of key generation operations completed per second.
signs/s	Number of signing operations completed per second.
verifies/s	Number of verification operations completed per second.

Classical EC and Ed signature result files contain only signing and verification metrics, without key-generation metrics.

The following table describes the metrics collected for PQC/Hybrid-PQC KEMs and classical XDH algorithms during TLS speed testing:

Metric	Description
keygen (s)	Average time in seconds to generate a keypair.
encaps (s)	Average time in seconds to perform an encapsulation operation.
decaps (s)	Average time in seconds for decapsulation operation.
keygens/s	Number of key generation operations completed per second.
encaps/s	Number of encapsulation operations completed per second.
decaps/s	Number of decapsulation operations completed per second.

Classical ECDH result files instead report op (s) and op/s, representing the average time and throughput for shared-secret derivation.

When running the TLS benchmarking script (pqc_tls_performance_test.sh), all performance data is initially stored as unparsed output. This includes both handshake and speed test results. After testing, the parsing script processes this raw data into structured CSV files, including calculated averages across test runs.

Data Type	State	Description	Location (relative to test_data/)
TLS Handshake	Un-parsed	Raw .txt outputs from OpenSSL s_time tests for PQC, Hybrid-PQC, and classical combinations.	up_results/tls_performance/machine_x/handshake_results/{pqc/hybrid/classic}
TLS Handshake	Parsed	Per-run PQC and Hybrid-PQC CSVs organised by signing algorithm.	results/tls_performance/machine_x/handshake_results/{pqc/hybrid}/{signature}
TLS Handshake	Parsed	Combined per-run and averaged CSVs for all classical signature/group/ciphersuite combinations.	results/tls_performance/machine_x/handshake_results/classic/
TLS Handshake	Parsed (Base)	Combined CSVs aggregating all PQC/Hybrid-PQC signature/KEM combinations for each run.	results/tls_performance/machine_x/handshake_results/{pqc/hybrid}/base_results
TLS Speed	Un-parsed	Raw .txt outputs from OpenSSL speed tests for PQC, Hybrid-PQC, and classical algorithms.	up_results/tls_performance/machine_x/speed_results/{pqc/hybrid/classic}
TLS Speed	Parsed	Per-run CSVs and averages with cryptographic operation timings and throughput.	results/tls_performance/machine_x/speed_results/{pqc/hybrid/classic}
Parsed Averages	Parsed	Averaged handshake and speed results across test runs.	Stored alongside parsed result files in results/tls_performance/machine_x/

Where machine_x is the Machine-ID number assigned to the results when executing the testing scripts. If no custom Machine-ID is assigned, the default ID of 1 will be set for the results.

Raw classical handshake files use a different naming convention to PQC/Hybrid-PQC files:

tls_handshake_classic_(run-number)_(signing-alg)_(key-exchange-group)_(ciphersuite).txt.

Parsing combines these into classic_results_run_(run-number).csv and produces classic_results_avg.csv; both CSV formats retain separate Signing Algorithm, Key Exchange Group, and Ciphersuite columns.

Raw classical speed files are named tls_speed_classic_sig_(run-number).txt and tls_speed_classic_key_exchange_(run-number).txt. Parsing separates their OpenSSL result tables into sig_ec, sig_rsa, key_exchange_ecdh, and key_exchange_xdh per-run CSVs, with a corresponding _avg.csv file for each category.