Developing Regression Tests - idaholab/HYBRID GitHub Wiki
Dymola Regression Test Needs
For the automatic regression test system to properly test the downloaded library the proper NHES library must be loaded automatically by dymola in the dymola.mos file located at C:\Program Files\Dymola 2020\insert\dymola.mos for example. To properly run the tests the NHES and the TRANSFORM libraries need to be automatically loaded by Dymola upon startup. This can be accomplished by adding to the Dymola.mos until it looks something like:
RunScript("$DYMOLA/insert/displayunit.mos", true);
definePostProcessing("SDF output", "Convert result file to SDF format", "Modelica.Utilities.System.command(\"\\\"%DYMOLA%/bin/dsres2sdf\\\" %RESULTFILE%.mat %RESULTFILE%.sdf\")");
openModel("C:\Users\FRICKL\Desktop\TRANSFORM3_20_2020\TRANSFORM-Library\TRANSFORM\package.mo"); //Loads Transform package.mo
openModel("C:\msys64\home\FRICKL\hybrid_devel\hybrid\models\NHES\package.mo"); //Loads NHES package from hybrid directory
cd("C:\Users\FRICKL\Desktop\TESsystem"); //Place where all the Dymola runs will occur.
Having the Dymola.mos file written like this will allow Dymola to automatically load all the needed libraries for Regression testing when completed from the Regression Test Harness.
Test Size
- Test size should not be greater than 2MB in size.
Test Creation
To create a regression test once a user develops an example test in the Dymola NHES library can be accomplished through a couple of settings. In the Dymola simulation setup tab in the output tab uncheck the store at variable events box. Then click store in model button and check the output box, click ok, then click ok again, then Save the model. Example settings are shown below.
In the simulateModel command one of the following two flags is required. Either "numberOfIntervals" or "OutputInterval". numberOfIntervals tells dymola how many output intervals to make. OutputInterval tells dymola at what timestep interval should an output be present for comparison. The .mat file in the gold folder will need to be run using the same simulateModel command that is present in the .mos file being created.
These can be selected in the Simulation Setup tab of the Dymola GUI and should carry down to the command you copy and paste in the .mos file. An example is shown below of the simulation setup tab. pictures/Regression_picture.png
Then run the simulation, (ideally a test should take less than 100 seconds). On the simulation tab in the command line copy the simulation command. Example below:
simulateModel("NHES.Systems.EnergyManifold.SteamManifold.Examples.SteamManifold_Test", stopTime=100, numberOfIntervals=100, method="Esdirk45a", resultFile="SteamManifold_Test");
This command should then be added to a file and named Test_Example.mos. The command can be found in the Simulation Setup tab of the Dymola GUI once you hit simulate.
Then in folder /path/to/hybrid/hybrid/tests/dymola_tests create a folder named Test_YourModel.
create a gold folder in the new folder, drop the .mat file from your simulation that is named resultFile="SteamManifold_Test from your simulateModel command in the gold folder. Then in the main Test_YourModel folder drop the Test_Example.mos file and create a tests file open it up and place the following in it:
[Tests]
[./]
type = 'HYBRIDTester'
input = 'Test_Example.mos'
workingDir = '.'
output = 'SteamManifold_Test.mat'
dymola_mats = 'SteamManifold_Test.mat'
rel_err = 0.001
[../]
[]
where SteamManifold_Test.mat should be your result .mat file name, rel_error is the amount of error allowed between the gold file and the regression test output, and Test_Example.mos is the run script created.
Advanced Test File Options Utilized for complex models.
For complex models the initialization phase of a simulation can take the Modelica solvers a significant amount of time to find an initialization point. This occurs due to the highly nonlinear nature of the underlying physical equations. A way to avoid such situations is to provide a restart file to bypass the initialization phase of the simulation. A restart file is automatically created at the end of each simulation as the dsfin.txt file created in the folder where the simulation is run. This file includes the final values of the previous simulation from which the new model can restart. Move this file to the gold folder for your new testing system.
Once this file is created it can then be loaded automatically via the continue button in Modelica under the Simulation Tab. Select Continue -> Import Initial -> dsfin.txt
Then once the dsfinal.txt file is loaded go into the Setup tab and move the time back to start from zero and the end time to the desired simulation point for the test. This is necessary since Dymola assumes the user wants to restart the simulation from where it ended in time as well. This is not the case for the test. Instead the goal is to skip the initialization phase of the simulation and provide a clean solution from which to compare. To make this time change automatically, the user should change the dsfinal.txt file line 9 to the desired time.
Simulate this model and save the result file, in this example “NSSS_Test.mat” and place it into the gold folder of the testing system. Additionally, copy and paste the simulateModel command that is in the Dymola gui as the last line of your .mos script file for the test. The first two lines should be translateModel to make sure the right model is loaded into the equation set, followed by the importInitial command that loads all the values into the translated Model. The final command should be the simulateModel command.
The .mos file should look something like what is shown below.
translateModel("NHES.Systems.PrimaryHeatSystem.Westinghouse4LoopPWR.Examples.NSSS_Test");
importInitial("./gold/dsfinal.txt");
simulateModel("NHES.Systems.PrimaryHeatSystem.Westinghouse4LoopPWR.Examples.NSSS_Test", stopTime=10000, numberOfIntervals=250, method="Esdirk45a", resultFile="NSSS_Test");
Different Dymola Versions
In the case of some complicated models, it is possible that updates to the Modelica language and to Dymola's solving processes and capabilities will trigger failures in the testing harness even though >99% of the model simulation appears identical. This issue has been noted significantly in primarily large fluid models and the transition between Dymola 2021x (and before) and Dymola 2022. There are a couple of added steps to correct this issue. First, check the simulation output manually to make sure that the overall behavior has not changed (the test output should list variables that are failing the diff test). If the simulation output is reasonably the same, only altering in minor ways (same macro behavior but slightly altered micro behavior), the testing harness can check multiple .mat files in the gold folder. Generate the reference file in the version of Dymola that is currently failing in the test, and insert that .mat file in addition to the originally created file. The testing harness logic now states that if any files match the simulation output, then the test should pass.