Preparing the General Testing Envrionment - crt26/PQC-LEO GitHub Wiki

To understand the steps that should be taken to prepare the testing environment for energy usage testing, it is important to first understand the general process that is followed when using the automated testing scripts. The below image details the general steps taken during testing to signal and coordinate the testing and energy data collection process between the testing machine and the collection machine.

The control signalling and coordination process between the testing machine and the collection machine is facilitated through the use of a communication channel between the two machines. The communication channel is used to indicate when testing is starting and stopping, which in turn signals when energy data collection should start and stop on the collection machine.

Currently the control signalling process can utilise either a network-based communication channel or a serial communication channel. The following sub-sections provide an overview of the general steps that should be taken to support the use of each type of communication channel for control signalling when using the automated testing scripts.

To use network-based communication for control signalling, the testing machine and the collection machine must have a network connection that allows them to communicate with each other. Currently, the control signalling process uses UDP packets to send signals, with support for TCP packets to be added in the future.

By default, the following ports are used for network-based control signalling when using the automated testing scripts and should be open and available for use on both the testing machine and the collection machine:

It is possible to alter the default ports used. Pass --use-custom-eng-control-ports to the computational or TLS handshake testing script, or --use-custom-net-control-ports to the TLS operations energy testing script, and provide the custom ports when prompted.

During operation, the user will be prompted to enter the following information to configure the network-based communication channel when calling the automated testing scripts:

• The local IP address to be used for listening for incoming control signals
• The remote IP address to be used for sending control signals to

A bidirectional serial connection must be established between the testing machine and the collection machine to allow for control signalling to take place. The serial connection must also be configured to use a baud rate of 115200, with support for other baud rates to be added in the future. There are several methods by which this connection can be established, including using a USB-to-UART adapter or a direct UART connection between the two devices. The specific method used will depend on the hardware requirements of the devices being used. The tools available within PQC-LEO for serial communication are limited to Linux operating systems only.

Please refer to documentation specific to your device on enabling UART support and establishing a serial connection between the testing machine and the collection machine for details on how to set up a serial connection for control signalling. During operation, the user will be prompted to select which serial port to use during testing.

Before using the automated testing scripts for energy usage evaluations, the testing environment must be properly prepared and configured. The following section provides an overview of the general steps that should be taken to prepare the testing environment for energy usage testing.

It is important to note that if using the energy_collector_tools outside of the provided automation scripts, the way in which the testing environment is prepared and configured may differ. Please refer to the relevant energy meter and collection tools documentation for details on how to prepare and configure the testing environment when using the energy collector tools outside of the provided automation scripts. Links to the relevant documentation can be found in the Additional Documentation section of this page.

This section provides the general steps to configure the environment for testing, so please review the following sub-sections carefully to understand the necessary steps to prepare the testing environment for energy usage testing using the automated testing scripts. This section provides guidance on the following steps:

• Details on the standard testing environment device setup
• Guidance on setting up the energy meter for data collection
• Guidance on preparing the collection machine for data collection

When using the automated testing scripts and collection tools provided by PQC-LEO, the testing environment will follow this general structure:

How the energy meter is accessed and utilised will differ depending on the specific energy meter being used and the way in which it is set up. For details on how to set up and configure supported energy meters for use with the automated testing scripts, please refer to the Supported Energy Meters page.

When using the automated testing scripts, the energy meter must be configured so that the collection machine can access the energy usage metrics collected during testing. The setup process depends on the specific energy meter being used and how it is connected.

For instructions on configuring supported energy meters for use with the automated testing scripts, refer to the Supported Energy Meters page.

After the energy meter is properly connected to the collection machine, the following script located in the scripts/test_scripts directory can be used to start the energy usage data collection process on the collection machine:

./energy_metric_collector.sh

This script provides an automated interface for the collector program included within the PQC-LEO energy_collector_tools. It is possible to use the collector program directly without using the energy_metric_collector.sh script, but the script provides a more user-friendly interface for easily starting and configuring the collection process. For details on how to use the collector program directly, please refer to the energy collector tools usage guide page.

Once the script has been called, the user will then be prompted to enter in the following parameters to configure the collection process:

• Which test collection type is being performed (out of the supported energy usage testing types)
• Whether the results should have a custom Machine-ID assigned to them
• The number of test runs that will be performed

After these parameters have been entered, the script will prompt the user to configure the communication channel that will be used for control signalling between the testing machine and the collection machine. For details on how to set up the communication channel, please refer to Enabling Support for Test Control Signalling.

Finally, after the communication channel has been configured, the script will prompt the user to select which supported energy meter type will be used for data collection. After the meter type has been selected, the remaining device-specific communication setup prompts are shown (for example selecting the serial port used by the meter). This process may differ between devices, so please refer to the relevant energy meter documentation for details on configuration steps.

After the testing environment has been properly prepared and configured, the automated energy usage testing can be initiated. The process will differ depending on the specific type of energy usage testing being performed.

Please refer to the following guides for details on how to perform each of the supported testing types:

• PQC Computational Energy Usage Testing Guide
• TLS Handshake Energy Usage Testing Guide
• TLS Operations Energy Usage Testing Guide

Once testing has been completed, the collected energy usage metrics will be available on the collection machine. Please refer to the Outputted Results section of the relevant testing guide for details on where the collected energy usage metrics data is stored and how to access it.

The following advanced testing options can be enabled by passing the relevant command line arguments to the automated testing scripts:

• Baseline Measurement Duration
• Custom System Performance State Configuration
• Custom Control Signalling Ports

By default, the computational, TLS handshake, and TLS operations energy testing tools will gather a baseline measurement for 60 seconds before beginning testing. This duration can be configured by passing --energy-baseline-duration=<TIME> when calling the testing machine script. The specified duration must be a positive whole number of seconds. No additional flag is required on the collection machine, as it will continue recording until the testing machine sends the stop signal.

Examples of using this flag to gather a baseline measurement for 120 seconds in each testing category are as follows:

./pqc_performance_energy_test.sh --energy-baseline-duration=120
./pqc_tls_performance_test.sh --enable-energy-testing --energy-baseline-duration=120
./tls_operations_energy_test.sh --energy-baseline-duration=120

When configuring the baseline duration for TLS handshake energy usage testing, the --energy-baseline-duration flag must be passed alongside --enable-energy-testing when calling the client machine script. The script will report an error if energy usage testing has not been enabled. The handshake test duration and energy meter polling interval are configured separately.

The baseline measurement duration can also be configured when using the standalone comp_energy_tester tool. Please refer to the Computational Energy Tester Tool Usage Guide page for further information on using this functionality.

By default, the testing scripts will automatically configure the system performance state for energy usage testing, which includes setting the CPU performance to maximum and selecting a specific target CPU core for testing processes. However, it is possible to manually configure the system performance state settings by passing the --use-custom-sys-state flag to the testing script and configuring the desired settings when prompted. This allows for more granular control over the system performance state settings during energy usage testing, which can be useful for testing under different system performance conditions.

An example of using this flag when launching the testing script is as follows:

./pqc_performance_energy_test.sh --use-custom-sys-state

By default, the automated testing scripts for energy usage evaluations use set UDP ports for network-based control signalling between the testing machine and the collection machine.

However, it is possible to alter the default ports used for control signalling. Pass --use-custom-eng-control-ports to the computational or TLS handshake testing script, or --use-custom-net-control-ports to the TLS operations energy testing script, and provide the custom ports when prompted.

An example of using this flag when launching the testing script is as follows:

./pqc_performance_energy_test.sh --use-custom-eng-control-ports

• Supported Energy Meters
• Integrating New Energy Meters
• Supported Algorithms
• Project Tools
• Comp Energy Usage Tester Tool Guide
• Energy Collector Tool Guide