Execution and Evaluation of Simulations

Executing and Evaluating the Simulations

From the main entry point of the EvaluationFramework, i.e. by executing the steps as explained here, the following actions take place:

1. The C#-class "EvaluationData" automatically reads the two .JSON-files defining the Evaluation Space using Newtonsoft Json.NET
2. Using Eric Lippert's source code for calculating a Cartesian Product with LINQ from his blog, a list of all combinations of EvaluationParameters is created
3. Simultaneously, one "SimulationInstanceHandler" object is created for each combination of EvaluationParameters (see below)
4. Finally, every combination of EvaluationParameters is send over to the Unity executable, and the previously defined ObjectiveValues are received

Parallel Execution of Simulations

The following figure depicts how the parallel execution of simulations is realised in the EvaluationFramework:

1. As previously mentioned, one object of the "SimulationInstanceHandler"-class is created for each combination of EvaluationParameters
2. Executing simulations in parallel is realised through the .NET Task Parallel Library (TPL)
3. Using Daniel Schroeder's solution for limiting the number of tasks running in parallel in C# from his blog, the number of executables running in parallel is automatically limited to the number of CPU cores of the underlying hardware
4. Each "SimulationInstanceHandler"-object then stores the respective ObjectiveValues retrieved from the executable
5. After all combinations of EvaluationParameters have been simulated, all parameters and results are stored into the file "Results.txt"

Interprocess Communication

The communication of EvaluationParameters and ObjectiveValues between any "SimulationInstanceHandler"-object and the Unity executable is achieved through .NET Named Pipes:

For each set of parameters, one pipe is opened. On creation, each "SimulationInstanceHandler"-object got a unique ID assigned. Consequently, each "SimulationInstanceHandler"-object has two pipes, one for EvaluationParameters and one for ObjectiveValues, where each pipe has the same unique ID assigned, to ensure correct communication between the Unity executable and the .NET solution, while still being able to run multiple simulations in parallel.
In order for the Unity executable to successfully read the EvaluationParameters arriving through the "EvaluationParametersX"-pipe, the order of parameters sent to the executable exactly match the order of definition of EvaluationParameters earlier during the process. The Unity executable is aware of this order of parameters, since it stored all information on EvaluationParameters in a Component of the "EvaluationFramework" GameObject (see here).

If ROS is incorporated into the Evaluation Framework as described here, an additional Pipe is opened for the communication of ROSParameters from the .NET solution to the Unity executable:

Asynchronous Execution of Interprocess Communication

The following figure schemes the sequence of asynchronous operations when looking at one specific "SimulationInstanceHandler"-object communicating with one Unity executable:

1. Create a TPL task to open a pipe to send EvaluationParameters to the executable. Inside this task, asynchronously wait for a connection on the pipe
   1. When a connection is established, write each EvaluationParameter into the pipe and dispose the task.
2. Create a TPL task to open a pipe to receive ObjectiveValues from the executable. Inside this task, asynchronously wait for a connection on the pipe
   1. When a connection is established, read all ObjectiveValues from the pipe and dispose the task.
3. Synchronously call the executable (while both pipes are asynchronously waiting for a connection) and block enclosing task execution until the executable process is disposed again
4. Synchronously wait for the two pipe tasks to be disposed correctly
5. Store results read from the ObjectiveValue pipe and dispose the enclosing task

---

© Siemens AG, 2020

Author: Michael Dyck ([email protected])