Basic Idea - siemens/evaluation-framework GitHub Wiki

Basic Idea

The Evaluation Framework is a set of C# and Python libraries enabling the easy and efficient evaluation of Unity simulation models.

Idea - Virtual Prototyping

The idea of the Evaluation Framework is based on the concept of Virtual Prototyping, see e.g. Wikipedia or this publication. More specifically, the Evaluation Framework serves as a tool for design space exploration in Virtual Prototyping.

In Virtual Prototyping, (usually 3D) simulation models of e.g. mechatronic systems are examined and evaluated with the goal of optimizing the production costs, time-to-market, and the product itself. Without the need for physical prototypes, different product designs from different engineering specialities (mechanics, electronics, control, software development) can be tested and evaluated early in the production process. In order to do so, the following parameters have to be specified:

  • Design alternatives of the product ot be tested, e.g. design of mechanical and electrical components, controller structures, etc.. In the Evaluation Framework these parameters are referred to as EvaluationParamters
  • Decision criteria the product is evaluated against, e.g. product weight for specific mechanical designs, electric energy consumption with different electrical components, stability of various controllers, etc.. In the Evaluation Framework these parameters are referred to as ObjectiveValues

Both the EvaluationParameters and ObjectiveValues together define the complete Evaluation Space. The following GIF shows an example of possible EvaluationParameters and ObjectiveValues in the case of a simulation model of a plane.

Hence, in Virtual Prototyping, countless design alternatives, i.e. EvaluationParameters, of simulation models of mechatronic systems (or other products) are simulated and the specified decision criteria, i.e. ObjectiveValues, are used to evalute - and finally optimize - the performance of the respective product designs.

Implementation - 3 Evaluation Framework Components

The Evaluation Framework realizes this idea of performing Virtual Prototyping through three distinct software components. All of these components can also be used individually.

  • Definition of Evaluation Space: In Unity it is very easy to build physics simulations, and hence Unity serves as a great tool for simulation of mechatronic systems. The first component of the Evaluation Framework, i.e. EvaluationFrameworkUnity EvaluationFrameworkROSUnity, is a Unity addon that allows users to intuitively and easily define EvaluationParamters and ObjectiveValues from any Unity simulation model. In the end, the specified EvaluationSpace is automatically saved into two .json-files and the Unity simulation is built into a standalone application (executable).

  • Execution and Evaluation: The second component of the Evaluation Framework is implemented as a .NET solution in C#, see EvaluationFramework and EvaluationFrameworkROS. This component efficiently simulates the Unity project for all combinations of the previously defined EvaluationParamters. In other words, the .NET solution repeatedly calls the executable of the Unity simulation and performs a brute-force evaluation of the complete parameter space and retrieves the ObjectiveValues from the executable. In order to speed up this process, the underlying hardware is used to its maximum capacity, as one simulation can run per CPU core, thus parallelizing the brute-force evaluation. Finally, the results are stored in a table-formatted .txt-file containing all EvaluationParameter values and the corresponding ObjectiveValue results.

  • Point Cloud Visualization: The third component of the Evaluation Framework visualizes the evaluation results obtained from the second step in an interactive Design Space Exploration GUI. The corresponding source code can be found in the VisualizationGUI folder. The GUI has multiple interactive elements that allow the user to inspect the Evaluation Space from different perspectives, compare ObjectiveValues for different combinations of EvaluationParameters and identify conflicting design alternatives and product solutions.

For a full example on using the Evaluation Framework, please have a look at this application example.


© Siemens AG, 2020

Author: Michael Dyck ([email protected])