The test automation related files are contained AD-EYE_Core/AD-EYE/TA/.

Running a test automation scenario

SSH needs to be enabled on the Ubuntu computer. This can be done using the command sudo service ssh start (if this command does not work a package for SSH must be installed: sudo apt-get install openssh-server). The windows computer needs to have a file containing the parameters for the SSH connection. See AD-EYE_Core/AD-EYE/TA/Configurations/SSHConfigTemplate.csv to have an example.

The test automation can then be ran by using the function TA defined in AD-EYE_Core/AD-EYE/TA/TA.m. This function requires a TAOrder file that describes what should be ran (an example can be found in AD-EYE_Core/AD-EYE/TA/Configurations). It can also take two optional arguments that describe the index of the first and last experiments from the TAOrdeer file that should be ran.

If the test automation is interrupted, the ROS processes might still be running on the Ubuntu computer. The following commands can be run in an Ubuntu terminal:

 rosnode kill -a
 killall -9 rosmaster

General process overview

Windows side

The concept of the test automation is to run multiple simulations in a automated way. Prescan allows test automation with or without rebuilding the Prescan experiment.

The test automation with rebuild offer more maneuverability in the parameters to tune and is therefore the one used. This allows to modify the parameters related to Prescan (such as sensors parameters for examples). Through Matlab code, the Prescan experiment is duplicated in the Results folder of the template experiment. It is rebuilt, its Simulink compilation sheet is regenerated and its Simulink model is modified according to the parameters. This process is done for each of the teat automation configuration (column in the TAOrder file).

Simulink constants can be modified using Matlab code before running the Simulink experiment.

Linux side

To modify the parameters on the ROS side a set of launch files templates is used. In those templates each parameter points to a path on the ROS parameter server. From the Matlab side (TA.m) the wanted values are loaded on the ROS parameter server on the corresponding paths and, then, the template files are modified to replace the path by the value pointed by that path (through AD-EYE_Core/AD-EYE/ROS_Packages/src/AD-EYE/sh/launchTemplateModifier.sh).

The templates are in AD-EYE_Core/AD-EYE/ROS_Packages/src/AD-EYE/template_launch_files while the modified templates are in AD-EYE_Core/AD-EYE/ROS_Packages/src/AD-EYE/modified_launch_files.

The ROS side is launched using AD-EYE_Core/AD-EYE/ROS_Packages/src/AD-EYE/sh/managerFileLaunch.sh which starts the usual manager while setting the parameter /test_automation to true.

Test automation parameters values

The values used for the test automation are defined in csv files or Excel sheets. Those files are all located in the AD-EYE_Core/AD-EYE/TA/ folder.

File name	Description
TAOrder	Describes which map and which configuration files to load
AutowareConfig	Describes the parameters that will be placed in the ROS launch files
SimulinkConfig	Simmulink parameters (contains the goal)
SSHConfig	Parameters to connect to the Ubunutu computer through SSH
TagsConfig	Text file containing the tags for the command line call to PreScan

Editing a test automation scenario

Changing one of the previous files allows to modify the test automation parameters. TAOrder contains a general description of what the test automation will do while AutowareConfig and SimulinkConfig contain the values of each parameter.

To add test automation to a PreScan world or to add new variables test automation must first be activated by clicking on Experiment, General Setting and Test Automation. This menu allows to enable test automation on different actors and on their sensors.

Once test automation has been activated, a label must be added. Right click on an actor and click on Test Automation.

Monte Carlo Sampling

A wrapper was implemented to only run TA automation with a uniformly sampled subset. The Matlab function called MonteCarloTA takes as an input the TAOrder file that contains the full parameter set and samples it to generate the sampledTAOrder subset. This table containing the samples is then written as TAOrder_Monte_Carlo.xlsx and TA is called with that file.