PQC TLS Performance Metrics - crt26/pqc-evaluation-tools GitHub Wiki

OQS-Provider PQC TLS Performance Metrics

The OQS-Provider TLS performance testing captures benchmarking data for PQC and Hybrid-PQC algorithms integrated into the OpenSSL 3.4.1 library. It evaluates both their performance within the TLS 1.3 handshake protocol and their cryptographic operation speed when executed directly through OpenSSL. This testing provides valuable insight into how PQC schemes perform in real-world security protocol scenarios. Additionally, TLS handshake metrics are gathered using classical digital signature algorithms and cipher suites to establish a performance baseline for comparison with PQC and Hybrid-PQC results.

As part of the automated TLS testing, two categories of evaluations are conducted:

• TLS Handshake Testing - This simulates full TLS 1.3 handshakes using OpenSSL’s s_server and s_time tools, evaluating both standard and session-resumed connections.

• TLS Speed Testing - This uses the OpenSSL s_speed tool to benchmark the algorithm’s low-level operations, such as key generation, encapsulation, signing, and verification.

TLS Handshake Testing

The TLS handshake performance tests measure how efficiently different PQC, Hybrid-PQC, and classical algorithm combinations perform during the TLS 1.3 handshake process. These tests are executed using OpenSSL's built-in benchmarking tools (s_server and s_time) with the OQS-Provider integration.

Each test performs the TLS handshake for a given digital signature and KEM algorithm combination (digital signature) as many times as possible for a set time window, both with and without session ID reuse, to evaluate the impact of session resumption on performance.

The table below describes the performance metrics gathered during this testing:

Metric	Description
Connections in User Time	Number of successful TLS handshakes completed during CPU/user time. Reflects algorithm efficiency per CPU second.
Connections per User Second	Handshake rate per CPU second. Indicates performance under ideal CPU conditions.
Real-Time	Total wall clock time elapsed, including system I/O and process delays.
Connections in Real Time	Number of handshakes completed in actual wall time. Useful for real-world performance assessment.
Connections per User Second (Session Reuse)	Handshake rate per CPU second with session ID reuse. Measures efficiency with session resumption.
Connections in Real Time (Session Reuse)	Handshakes per real-world time with session reuse. Reflects practical performance with resumed sessions.

TLS Speed Testing

TLS speed testing benchmarks the raw cryptographic performance of PQC and Hybrid-PQC algorithms when integrated into the OpenSSL library via the OQS-Provider. This is done using the OpenSSL s_speed tool, which measures the execution time and throughput of cryptographic operations for each algorithm.

The primary objective of this test is to gather the base system performance of the schemes when integrated into the OpenSSL library. The results provide insight into the algorithm’s standalone efficiency when running within OpenSSL, which can produce additional overhead compared to the performance tests provided by Liboqs.

Digital Signature Algorithm Metrics

The following table describes the metrics collected for digital signature algorithms during TLS speed testing:

Metric	Description
keygen (s)	Average time in seconds to generate a signature key pair.
sign (s)	Average time in seconds to perform a signing operation.
verify (s)	Average time in seconds to verify a digital signature.
keygens/s	Number of key generation operations completed per second.
signs/s	Number of signing operations completed per second.
verifies/s	Number of verification operations completed per second.

KEM Algorithm Metrics

The following table describes the metrics collected for Key Encapsulation Mechanism (KEM) algorithms during TLS speed testing:

Metric	Description
keygen (s)	Average time in seconds to generate a keypair.
encaps (s)	Average time in seconds to perform an encapsulation operation.
decaps (s)	Average time in seconds for decapsulation operation.
keygens/s	Number of key generation operations completed per second.
encaps/s	Number of encapsulation operations completed per second.
decaps/s	Number of decapsulation operations completed per second.

OQS-Provider Result Data Storage Structure

When running the OQS-Provider TLS benchmarking script (full-oqs-provider-test.sh), all performance data is initially stored as unparsed output on the client machine. This raw data is then processed using the Python parsing script to generate structured CSV files for analysis, including averaged metrics across multiple test runs.

Data Type	State	Description	Location
TLS Handshake	Un-parsed	Raw .txt outputs from OpenSSL s_time tests for PQC, Hybrid-PQC, and Classic algorithm combinations.	test-data/up-results/oqs-provider/machine-X/handshake-results/{pqc/hybrid/classic}
TLS Handshake	Parsed	Per-run CSVs with extracted handshake metrics (PQC, Hybrid, Classic), separated by each digital signature.	test-data/results/oqs-provider/machine-X/handshake-results/{pqc/hybrid/classic}/{signature-name}
TLS Handshake	Parsed (Base)	Full combined metrics for all digital signature and KEM combinations in a single CSV for each run.	test-data/results/oqs-provider/machine-X/handshake-results/{pqc/hybrid}/base-results
TLS Speed	Un-parsed	Raw .txt outputs from openssl speed tests for PQC and Hybrid-PQC algorithms (digital signature and KEM).	test-data/up-results/oqs-provider/machine-X/speed-results/{pqc/hybrid}
TLS Speed	Parsed	Cleaned CSVs with cryptographic operation timings and throughput per algorithm.	test-data/results/oqs-provider/machine-X/speed-results/
Parsed Averages	Parsed	Averaged handshake/speed metrics across test runs.	Same as parsed result directories (results/oqs-provider/machine-X/)