PQC TLS Performance Metrics - crt26/PQC-LEO GitHub Wiki

Collected Performance Metrics

The TLS performance testing suite benchmarks PQC, Hybrid-PQC, and classical algorithm configurations available through both OpenSSL's native support and the OQS-Provider. As of OpenSSL 3.5.0, PQC algorithms are supported through both sources, and the suite is designed to evaluate performance consistently across the full range of available implementations. It measures performance within the TLS 1.3 handshake protocol and the execution speed of cryptographic operations directly through OpenSSL. This provides insight into how PQC schemes perform in real-world security protocol scenarios. Classical digital signature algorithms and ciphersuites are also tested to establish a performance baseline for comparison with PQC and Hybrid-PQC configurations.

As part of the automated TLS testing, two categories of evaluations are conducted:

• TLS Handshake Testing - This simulates full TLS 1.3 handshakes using OpenSSL’s s_server and s_time tools, evaluating both standard and session-resumed connections.

• TLS Speed Testing - This uses the OpenSSL s_speed tool to benchmark the algorithm's low-level operations, such as key generation, encapsulation, signing, and verification.

TLS Handshake Testing

The TLS handshake performance tests measure how efficiently different PQC, Hybrid-PQC, and classical algorithm combinations perform during the TLS 1.3 handshake process. These tests are executed using OpenSSL's built-in benchmarking tools (s_server and s_time).

Each test performs the TLS handshake for a given digital signature and KEM algorithm combination (digital signature) as many times as possible for a set time window, both with and without session ID reuse, to evaluate the impact of session resumption on performance.

The table below describes the performance metrics gathered during this testing:

Metric	Description
Connections in User Time	Number of successful TLS handshakes completed during CPU/user time. Reflects algorithm efficiency per CPU second.
Connections per User Second	Handshake rate per CPU second. Indicates performance under ideal CPU conditions.
Real-Time	Total wall clock time elapsed, including system I/O and process delays.
Connections in Real Time	Number of handshakes completed in actual wall time. Useful for real-world performance assessment.
Connections per User Second (Session Reuse)	Handshake rate per CPU second with session ID reuse. Measures efficiency with session resumption.
Connections in Real Time (Session Reuse)	Handshakes per real-world time with session reuse. Reflects practical performance with resumed sessions.

TLS Speed Testing

TLS speed testing benchmarks the raw cryptographic performance of PQC, Hybrid-PQC, and classical algorithms when integrated into OpenSSL for both natively supported algorithms and those provided by the OQS-Provider library. This is done using the OpenSSL s_speed tool, which measures the execution time and throughput of cryptographic operations for each algorithm.

The primary objective of this test is to gather the base system performance of the schemes when integrated into the OpenSSL library. The results provide insight into the algorithm's standalone efficiency when running within OpenSSL, which can produce additional overhead compared to the performance tests provided by the computational performance testing suite.

Digital Signature Algorithm Metrics

The following table describes the metrics collected for digital signature algorithms during TLS speed testing:

Metric	Description
keygen (s)	Average time in seconds to generate a signature key pair.
sign (s)	Average time in seconds to perform a signing operation.
verify (s)	Average time in seconds to verify a digital signature.
keygens/s	Number of key generation operations completed per second.
signs/s	Number of signing operations completed per second.
verifies/s	Number of verification operations completed per second.

KEM Algorithm Metrics

The following table describes the metrics collected for Key Encapsulation Mechanism (KEM) algorithms during TLS speed testing:

Metric	Description
keygen (s)	Average time in seconds to generate a keypair.
encaps (s)	Average time in seconds to perform an encapsulation operation.
decaps (s)	Average time in seconds for decapsulation operation.
keygens/s	Number of key generation operations completed per second.
encaps/s	Number of encapsulation operations completed per second.
decaps/s	Number of decapsulation operations completed per second.

TLS Performance Result Data Storage Structure

When running the TLS benchmarking script (full_tls_test.sh), all performance data is initially stored as unparsed output. This includes both handshake and speed test results. After testing, the parsing script processes this raw data into structured CSV files, including calculated averages across test runs.

Data Type	State	Description	Location (relative to test_data/)
TLS Handshake	Un-parsed	Raw .txt outputs from OpenSSL s_time tests for PQC, Hybrid-PQC, and Classic combinations.	up_results/tls_performance/machine_x/handshake_results/{pqc/hybrid/classic}
TLS Handshake	Parsed	Per-run CSVs with extracted handshake metrics, by signature algorithm.	results/tls_performance/machine_x/handshake_results/{pqc/hybrid/classic}/{signature}
TLS Handshake	Parsed (Base)	Combined CSVs aggregating all signature/KEM combinations for each run.	results/tls_performance/machine_x/handshake_results/{pqc/hybrid}/base_results
TLS Speed	Un-parsed	Raw .txt outputs from OpenSSL speed tests for PQC and Hybrid-PQC algorithms.	up_results/tls_performance/machine_x/speed_results/{pqc/hybrid}
TLS Speed	Parsed	Cleaned CSVs with cryptographic operation timings and throughput.	results/tls_performance/machine_x/speed_results/
Parsed Averages	Parsed	Averaged handshake and speed results across test runs.	Stored alongside parsed result files in results/tls_performance/machine_x/

Where machine_x is the Machine-ID number assigned to the results when executing the testing scripts. If no custom Machine-ID is assigned, the default ID of 1 will be set for the results.