Performance Data Parsing Scripts - crt26/pqc-evaluation-tools GitHub Wiki
Python Performance Parsing Scripts
Various Python files are included in the scripts/parsing-scripts directory and provide the automatic result parsing functionality. These include:
- parse_results.py
- liboqs_parse.py
- oqs_provider_parse.py
- results_averager.py
Unlike the other scripts within the repositories, this functionality can be performed in either Linux or Windows environments, assuming all required dependencies are present on the system. For further information on required dependencies, please refer to the Parsing Script Usage section in the main README file.
While several scripts are utilised for the result parsing process, only the parse_results.py is intended to be called manually. The main parsing script calls the remaining scripts depending on which parameters the user supplies to the script when prompted.
Please refer to the Performance Metrics Guide for a detailed description of the performance metrics that this project can gather, what they mean, and how these scripts structure the un-parsed and parsed data.
parse_results.py
This script acts as the main controller for the result-parsing processes. When called, the script will prompt the user for the various testing parameters such as:
- Which type of testing was performed
- How many machines were tested (facilitating the comparison between varying machine types)
- How many runs of testing were conducted on each of those machines
After gathering these parameters, the script will call the relevant sub-scripts to process the unparsed results in the test-data/up-results directory. The final output will store the parsed results in CSV format for the various tests performed, which can be found in the test-data/results directory.
It is important to note that if parsing results from multiple machines, the current limitations of the script require the same number of test runs to be performed. This will be addressed in future versions of the scripts. If parsing results from multiple machines where the types of tests conducted and the number of test runs do not match, it is best to perform the parsing of the data separately. Manual renaming can then be performed to fit the desired naming scheme.
liboqs_parse.py
This script contains functions for parsing un-parsed Liboqs benchmarking data, transforming unstructured speed and memory test data into clean, structured CSV files. It processes CPU performance results and memory usage metrics for each algorithm and operation across multiple test runs and machines. This script is not to be called manually and is only invoked by the parse_results.py script.
oqs_provider_parse.py
This script contains functions for parsing un-parsed OQS-Provider benchmarking data, transforming unstructured TLS handshake and speed test data into clean, structured CSV files. It processes performance metrics for PQC, hybrid-PQC, and classical algorithm combinations across multiple machines and test runs, outputting the results as structured CSV files. This script is not to be called manually and is only invoked by the parse_results.py script.
results_averager.py
This script provides utility classes to compute average performance metrics from parsed benchmarking results. It is used by both liboqs_parse.py and oqs_provider_parse.py to generate per-algorithm averages across multiple test runs. It handles memory and CPU performance metrics for Liboqs tests and handshake and speed metrics for OQS-Provider TLS tests. This script is not to be called manually and is only executed internally by the result parsing scripts.