Test Models - openmpp/openmpp.github.io GitHub Wiki
Home > Model Development Topics > Test Models
This topic contains detailed information about the OpenM++ test_models utility. test_models builds, runs, and compares results for different versions of the same model, or for the same model built in different ways or on different platforms.
- Introduction and overview
- Windows Quick start
- Linux or MacOS Quick start
- Concepts
- Arguments and Options
- Control - Files
- Output - Report
- Output - Files
- Example 1: Preparing the default run
- Example 2: A new OpenM++ release
- Example 3: A model code change (under construction)
- Example 4: When results differ (under construction)
- Example 5: Exercising a model in Debug (under construction)
- Example 6: A performance comparison
- Example 7: Using event trace output (under construction, some content present)
The test_models utility builds, runs, and compares results for different versions of the same model, or for the same model built on different platforms. It can process multiple models in a single invocation and can compare OpenM++ and Modgen versions of cross-compatible models. It can play a useful role during incremental model development, and for quality assurance.
test_models is efficient enough that it can be used routinely in model development workflow cycles, even for large models (using a small sized Default simulation). even multiple times per hour or more during active model development.
When used routinely to assess the effects of incremental model changes, test_models can help identify bugs in model logic. It highlights when model changes have unexpected effects in model outputs, by comparing all cells of all output tables between a Current run and a Reference run.
test_models can be used to compare outputs of ompp model versions across platforms. Such differences are rare, and are typically caused by errors in model logic (often due to bad C++ code in models, eg use of uninitialized variables, or accessing memory beyond array limits using index variables with bad values).
test_models works with the standard model folder structure illustrated in the sample models in the OpenM++ distribution. It organizes results herarchically in a subfolder test_models of the model folder. test_models organizes comparisons in two dimensions. The first dimension is {Reference, Current}. A user can tell test_models to move Current results to Reference results to create a new point of reference for subsequent output comparisons. The second dimension of comparison is the platform ‘flavour’ {modgen, ompp-win, ompp-linux, ompp-macos}. Arguments to test_models determine where model results will be stored and which comparisons will be performed after test_models does the requested runs. For speed and ease of use, each model output table is converted to three forms: full precision, rounded precision, and a summary 'digest' of the file contents. The use of rounded precision eliminates most (but not all) spurious differences, eg differences at 6-7th decimal place. The table digest makes it possible for test_models to determine whether two versions of a table differ at very high speed, without comparing the two tables cell-by-cell.
test_models can compare results of Modgen and ompp versions of models, despite large underlying differences in output storage schema and technology. It reads the database of each and converts output tables into a common comparable csv format.
test_models can be very useful for cross-compatible model conversion. One can break cross-compatible model conversion into atomic steps and use test_models after each atomic step to verify that Modgen results have not changed, even before an OpenM++ version of the model is buildable. The comparisons relevant during cross-compatible model conversion are
- Current(modgen) vs. Reference(modgen) - Modgen results should remain identical at each atomic change to model source, when build in Modgen.
- Current(ompp) vs. Current(modgen) – After the model code is cross-compatible, ie when the ompp version can be built with no compiler errors, the model can be run in both versions and model results should be identical, because the model specifications are identical (same model source code and same Default scenario). Sometimes they are not, in which case the tables which differ can sometimes provide a clue.
When tracking down the cause of an unexpected difference in two runs, it can cometimes be useful to simplify the Default scenario and run test_models again, to see which area of the model might responsible for the difference. For example, if all tables differ, but when one turns off immigration in the Default scenario all differences disappear, one knows that the immigration code is somehow responsible. It can also be useful to turn on event tracking in the model with a small simulated population. If test_models indicates that the two event tracking files are identical, but table results are different, then the cause could be a Modgen tabulation bug.
test_models is also useful for model development. One can break model development into small atomic steps to help identify unexpected changes in model outputs. If possible, one can structure model changes so that Default parameters and code changes should have no effect on outputs, eg by turning a new model option off in Default parameters. That can identify whether the new model code, when disabled using the new parameters, has unexpected effects on other parts of the model (it should not). The test_models comparisons typically used for model development are
- Current(ompp) vs. Reference(ompp) - Model results from incremental changes should affect only new tables or expected interactions among model components. Once verified, one can tell test_models to copy Current results to Reference results (using the
--newrefoption) to create a new Reference for the next atomic set of changes to model code.
If one prefers to work in the Modgen environment for model development, one can instead use
- Current(modgen) vs. Reference(modgen) - See notes above.
- Current(modgen) vs. Current(ompp) – From time to time, build the ompp version to identify if non cross-compatible code has crept into the model source, and rectify if so.
test_models also standardizes the detailed event trace outputs (if activated) in ompp and Modgen models so that they are comparable and readable. This can identify precisely when a simulation diverges in the Modgen and ompp versions of a model, or in (for example) two ompp versions of a model. This is useful for unexpected and hard-to-understand differences in simulations. If the time and event of earliest divergence in the two runs is insufficient to understand the cause in and of itself, that information provides what’s required to set conditional breakpoints for a parallel debugging session of the two runs, stepping through the simulation in each of the two debugging sessions, to identify the precise code location responsible for the divergence in the simulation in the two versions.
test_models also notes some tombstone information about each set of outputs, eg the modgen version. It also keeps a copy of the build log output and Default run log output. test_models can also report the elpased time of various steps, including model build and model run. A count of compiler warnings is also reported.
test_models can run and process a single model, or multiple models in subfolders of a parent folder. That can be useful for testing mutliple scenarios of a single model, using mutliple git clones inside a parent folder. It can also be useful to bulk test multiple models routinely, for example after a change in the OpenM++ version.
A 64-bit Windows executable version of test_models is distributed with OpenM++ at OM_ROOT/bin/test_models.exe, where OM_ROOT stands for the OpenM++ installation directory. A 32-bit version is at OM_ROOT/bin/test_models32.exe. The 32-bit version may be required for the Modgen flavour to work successfully using the executable version of test_models.
To test installation and operation of test_models, open a command prompt, change the current directory to OM_ROOT/bin, and type the command
test_models -v
Output should be similar to the following:
test_models version 2.1
test_models is written in the Perl language, and distributed with OpenM++ at OM_ROOT/Perl/test_models.pl. Most examples in this topic invoke test_models using the Perl interpreter from the OM_ROOT/Perl directory, eg
perl test_models.pl -v
On Windows, unless you have Perl and the required Perl components installed, invoke the executable version of test_models from the OM_ROOT/bin directory with a command like
test_models -v
or
test_models32 -v
From the OM_ROOT/bin directory, type the command
test_models -h
Output should be similar to the following:
test_models [-hmv] [long options...] model...
-m STR --models_root STR directory containing models (default is .)
--newref replace Reference results with Current
results
--noompp skip OpenM++ build and run
--nomodgen skip Modgen build and run
--nocomp skip flavour comparison
--allfiles report all different and orphaned files
--timing report elapsed time of steps
--nosteps skip reporting which step is being
performed
--config STR build configuration: debug or
release(default)
--mpi_processes INT build MPI version and run with n
processes (default 0, means no MPI)
--gencode keep a copy of the generated C++ code
--ini STR OpenM++ model ini file to pass to model
(in model root, default is
test_models.ini if present)
--clean remove all build files after run
--significant_digits INT significant digits (default 6)
--nocells disable fallback cell-by-cell
verification of differing tables and copy
of original data
-h --help report usage message and exit
-v --version report test_models version and exit
--windows_platform STR Windows platform: x64(default) or Win32
--modgen_platform STR Modgen platform: Win32(default) or x64on of unrounded versions
of tables
From the OM_ROOT/bin directory, type the command
test_models -m ../models RiskPaths
If Modgen is installed, use test_models32 instead of test_models. test_models will build and run both the OpenM++ and the Modgen versions of RiskPaths, and compare their results. Output should be similar to the following:
=========================
test_models 2.0
=========================
Testing: RiskPaths
modgen settings: version=12,1,3,0 (2019-12-19 20:31 GMT) platform=Win32 configuration=release
ompp-win settings: compiler=omc.exe (2021-05-28 02:28 GMT) platform=x64 configuration=release
RiskPaths: modgen: Build model and prepare Default scenario
RiskPaths: modgen: Run model using RiskPaths.ini
RiskPaths: modgen: Convert outputs (7 digits of precision)
RiskPaths: modgen: Create digests of current outputs
RiskPaths: modgen: No Reference outputs - create using Current outputs
RiskPaths: modgen: Current vs. Reference: 9 the same (of 9)
RiskPaths:
RiskPaths: ompp-win: Build and publish model and Default scenario
warning => RiskPaths: ompp-win: 10 build warning(s) - see RiskPaths/test_models/current/ompp-win/logs/build.log
RiskPaths: ompp-win: Run model using RiskPaths.ini
RiskPaths: ompp-win: Convert outputs (7 digits of precision)
RiskPaths: ompp-win: Create digests of current outputs
RiskPaths: ompp-win: No Reference outputs - create using Current outputs
RiskPaths: ompp-win: Current vs. Reference: 9 the same (of 9)
RiskPaths:
RiskPaths: Flavour comparisons:
RiskPaths:
RiskPaths: ompp-win vs. modgen: Reference: 9 the same (of 9)
RiskPaths:
RiskPaths: ompp-win vs. modgen: Current: 9 the same (of 9)
If Modgen is not installed, output will not include the Modgen portion nor the flavour comparison portions of the report.
test_models uses the Microsoft application msbuild.exe to build models. msbuild.exe is normally installed as part of Visual Studio installation, but the install location can vary. test_models attempts to determine the location of msbuild.exe but may not always succeed. If you encounter issues running test_models which seem related to msbuild.exe you can provide test_models the location explicitly using the environment variable MSBUILD_EXE, for example:
set MSBUILD_EXE=C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Current\Bin\msbuild.exe
in the command window before invoking test_models.
test_models is a Perl script distributed with OpenM++ at OM_ROOT/Perl/test_models.pl, where OM_ROOT stands for the OpenM++ installation directory.
To test installation and operation of test_models, open a command prompt, change the current directory to OM_ROOT\Perl, and type the command
perl test_models.pl -v
Output should be similar to the following:
test_models version 2.1
Depending on your operating system version and installation history, Perl may ask you to install missing Perl modules required by test_models.pl.
If so, it will name them explicitly when you invoke test_models.pl.
We do recommend to use cpanm for Perl modules installation. Typical scenario is:
cpan App::cpanminus # initialize cpanm, if not done before
cpanm Getopt::Long::Descriptive
cpanm Capture::Tiny
cpanm Config::Tiny
cpanm File::Copy::Recursive
cpanm File::Which
Above list of modules can be different and depends on your current Perl configuration, and on the version of test_models.
From the OM_ROOT/Perl directory, type the command
perl test_models.pl -h
Output should be similar to the following:
test_models [-hmv] [long options...] model...
-m STR --models_root STR directory containing models (default is .)
--newref replace Reference results with Current
results
--noompp skip OpenM++ build and run
--nomodgen skip Modgen build and run
--nocomp skip flavour comparison
--allfiles report all different and orphaned files
--timing report elapsed time of steps
--nosteps skip reporting which step is being
performed
--config STR build configuration: debug or
release(default)
--mpi_processes INT build MPI version and run with n
processes (default 0, means no MPI)
--gencode keep a copy of the generated C++ code
--ini STR OpenM++ model ini file to pass to model
(in model root, default is
test_models.ini if present)
--clean remove all build files after run
--significant_digits INT significant digits (default 6)
--nocells disable fallback cell-by-cell
verification of differing tables and copy
of original data
-h --help report usage message and exit
-v --version report test_models version and exit
--windows_platform STR Windows platform: x64(default) or Win32
--modgen_platform STR Modgen platform: Win32(default) or x64
From the OM_ROOT/Perl directory, type the command
perl test_models.pl -m ../models RiskPaths
Output should be similar to the following:
=========================
test_models 2.0
=========================
Testing: RiskPaths
ompp-linux settings: compiler=omc (2021-05-29 16:47 GMT) configuration=release
RiskPaths: ompp-linux: Build and publish model and Default scenario
RiskPaths: ompp-linux: Run model using RiskPaths.ini
RiskPaths: ompp-linux: Convert outputs (7 digits of precision)
RiskPaths: ompp-linux: Create digests of current outputs
RiskPaths: ompp-linux: No Reference outputs - create using Current outputs.
RiskPaths: ompp-linux: Current vs. Reference: 9 the same (of 9)
This subtopic describes key underlying concepts used in test_models.
- Run version: Current or Reference
- Run flavour: Windows, Linux, MacOS, Modgen
- Output comparison:
- Build options:
- Run options:
=== under construction ===
- Current or Reference
[back to concepts]
[back to topic contents]
=== under construction ===
- Windows, Linux, MacOS, Modgen
[back to concepts]
[back to topic contents]
=== under construction ===
[back to concepts]
[back to topic contents]
=== under construction ===
[back to concepts]
[back to topic contents]
=== under construction ===
[back to concepts]
[back to topic contents]
This subtopic describes the command line options and arguments of test_models, organized into sections. It also describes default test_models behaviour for each section.
- Syntax: The syntax of test_models arguments
- Models: The models to process
- Actions: The actions to perform on a model
- Verbosity: The level of report detail
- Build: How a model is built
- Run: How a model is run
- Comparison: How result are compared
- Informational: Informational options
- EventTrace: EventTrace options
A complete list of options is displayed by issuing the command
perl test_models.pl -h
test_models [-hmv] [long options...] model...
-m STR --models_root STR directory containing models (default is .)
--newref replace Reference results with Current
results
--noompp skip OpenM++ build and run
--nomodgen skip Modgen build and run
--nocomp skip flavour comparison
--allfiles report all different and orphaned files
--timing report elapsed time of steps
--nosteps skip reporting which step is being
performed
--config STR build configuration: debug or
release(default)
--mpi_processes INT build MPI version and run with n
processes (default 0, means no MPI)
--gencode keep a copy of the generated C++ code
--ini STR OpenM++ model ini file to pass to model
(in model root, default is
test_models.ini if present)
--clean remove all build files after run
--significant_digits INT significant digits (default 6)
--zero_fuzz NUM zero fuzz value (default 1e-15)
--nocells disable fallback cell-by-cell
verification of differing tables and copy
of original data
-h --help report usage message and exit
-v --version report test_models version and exit
--time_format STR time format for event trace (default
13.6f)
--modgen_id_offset INT offset adjustment to Modgen IDs (default
0)
--windows_platform STR Windows platform: x64(default) or Win32
--modgen_platform STR Modgen platform: Win32(default) or x64
[back to arguments and options]
[back to topic contents]
Arguments to test_models consist of a series of options followed by a list of models to process. Either or both may be empty, in which case default values are used. An option starts with -- followed immediately by the option name. Some options may be followed by an option value. Some options have a synonymous short form consisting of a single - followed imediately by a single letter.
[back to arguments and options]
[back to topic contents]
The --models_root option (short form -m) specifies the parent folder of one or more folders each of which contains a model. If not specified, the models root is the current working directory. Consider the following command, issued after setting the current working directory to OM_ROOT/Perl:
perl test_models.pl -m ../models NewCaseBased RiskPaths
On Windows without Perl installed, a similar command would be issued after setting the current working directory to OM_ROOT/bin:
test_models -m ../models NewCaseBased RiskPaths
The command instructs test_models to set the models root to the sister directory models in the OpenM++ distribution. That folder is OM_ROOT/models and contains several sample models distributed with OpenM++, each in its own subfolder, eg OM_ROOT/models/RiskPaths. The final two arguments instruct test_models to process the two models in the subfolders OM_ROOT/models/NewCaseBased and OM_ROOT/models/RiskPaths.
On Linux, output is similar to:
=========================
test_models 2.0
=========================
Testing: NewCaseBased, RiskPaths
ompp-linux settings: compiler=omc (2021-05-29 16:47 GMT) configuration=release
NewCaseBased: ompp-linux: Build and publish model and Default scenario
NewCaseBased: ompp-linux: Run model using test_models.ini
NewCaseBased: ompp-linux: Convert outputs (7 digits of precision)
NewCaseBased: ompp-linux: Create digests of current outputs
NewCaseBased: ompp-linux: No Reference outputs - create using Current outputs
NewCaseBased: ompp-linux: Current vs. Reference: 3 the same (of 3)
RiskPaths: ompp-linux: Build and publish model and Default scenario
RiskPaths: ompp-linux: Run model using RiskPaths.ini
RiskPaths: ompp-linux: Convert outputs (7 digits of precision)
RiskPaths: ompp-linux: Create digests of current outputs
RiskPaths: ompp-linux: No Reference outputs - create using Current outputs
RiskPaths: ompp-linux: Current vs. Reference: 9 the same (of 9)
On Windows with Modgen installed, output is similar to:
=========================
test_models 2.0
=========================
Testing: NewCaseBased, RiskPaths
modgen settings: version=12,1,3,0 (2019-12-19 20:31 GMT) platform=Win32 configuration=release
ompp-win settings: compiler=omc.exe (2021-05-28 02:28 GMT) platform=x64 configuration=release
NewCaseBased: modgen: Build model and prepare Default scenario
NewCaseBased: modgen: Run model using test_models.ini
NewCaseBased: modgen: Convert outputs (7 digits of precision)
NewCaseBased: modgen: Create digests of current outputs
NewCaseBased: modgen: No Reference outputs - create using Current outputs
NewCaseBased: modgen: Current vs. Reference: 3 the same (of 3)
NewCaseBased:
NewCaseBased: ompp-win: Build and publish model and Default scenario
warning => NewCaseBased: ompp-win: 3 build warning(s) - see NewCaseBased/test_models/current/ompp-win/logs/build.log
NewCaseBased: ompp-win: Run model using test_models.ini
NewCaseBased: ompp-win: Convert outputs (7 digits of precision)
NewCaseBased: ompp-win: Create digests of current outputs
NewCaseBased: ompp-win: No Reference outputs - create using Current outputs
NewCaseBased: ompp-win: Current vs. Reference: 3 the same (of 3)
NewCaseBased:
NewCaseBased: Flavour comparisons:
NewCaseBased:
NewCaseBased: ompp-win vs. modgen: Reference: 3 the same (of 3)
NewCaseBased:
NewCaseBased: ompp-win vs. modgen: Current: 3 the same (of 3)
RiskPaths: modgen: Build model and prepare Default scenario
RiskPaths: modgen: Run model using RiskPaths.ini
RiskPaths: modgen: Convert outputs (7 digits of precision)
RiskPaths: modgen: Create digests of current outputs
RiskPaths: modgen: No Reference outputs - create using Current outputs
RiskPaths: modgen: Current vs. Reference: 9 the same (of 9)
RiskPaths:
RiskPaths: ompp-win: Build and publish model and Default scenario
warning => RiskPaths: ompp-win: 10 build warning(s) - see RiskPaths/test_models/current/ompp-win/logs/build.log
RiskPaths: ompp-win: Run model using RiskPaths.ini
RiskPaths: ompp-win: Convert outputs (7 digits of precision)
RiskPaths: ompp-win: Create digests of current outputs
RiskPaths: ompp-win: No Reference outputs - create using Current outputs
RiskPaths: ompp-win: Current vs. Reference: 9 the same (of 9)
RiskPaths:
RiskPaths: Flavour comparisons:
RiskPaths:
RiskPaths: ompp-win vs. modgen: Reference: 9 the same (of 9)
RiskPaths:
RiskPaths: ompp-win vs. modgen: Current: 9 the same (of 9)
If no models are specified on the command line, test_models will process All subfolders of the models root.
[back to arguments and options]
[back to topic contents]
Unless instructed otherwise, test_models builds and runs all available 'flavours' of a model on the current operating system. On Windows, two flavours are available: OpenM++ and Modgen (if installed). On Linux and MacOS, the only available flavour is OpenM++. After building and running each flavour, test_models compares Current results to existing Reference results. If there are no Reference results, test_models creates them by copying Current results. After completing all flavours for a model, test_models compares results for other flavours (if present) to results for the OpenM++ flavour on the current operating system.
This default behaviour can be modified by the following command options:
-
--newrefFor each flavour in this invocation, discard all Reference results and replace them with Current results -
--noomppSuppress processing the OpenM++ flavour -
--nomodgenSuppress processing the Modgen flavour (only applies if invoked on Windows) -
--nocompSuppress reporting on differences between flavours
For example, on Windows, the command
test_models -m ../models --noompp --nomodgen RiskPaths
would process the RiskPaths model, but skip build and run of both Windows flavours (OpenM++ and Modgen). test_models would still compare results for any previously run flavours to OpenM++ Windows results (if present). Output might be similar to the following (or could be empty, if no other flavours were run previously):
=========================
test_models 2.0
=========================
Testing: RiskPaths
RiskPaths:
RiskPaths: Flavour comparisons:
RiskPaths:
RiskPaths: ompp-win vs. modgen: Reference: 9 the same (of 9)
RiskPaths:
RiskPaths: ompp-win vs. modgen: Current: 9 the same (of 9)
RiskPaths:
RiskPaths: ompp-win vs. ompp-linux: Reference: 9 the same (of 9)
RiskPaths:
RiskPaths: ompp-win vs. ompp-linux: Current: 9 the same (of 9)
In this invocation, the RiskPaths model was neither built nor run. test_models detected the presence of Current and Reference results for two other flavours (Modgen and OpenM++ on Linux), and compared them to results from OpenM++ on Windows which were present from a previous invocation of test_models. Incidentally, because test_models uses digests to compare results, comparing all results between two flavours is almost instantaneous, even for models with many tables or with very large tables.
[back to arguments and options]
[back to topic contents]
Unless instructed otherwise, test_models reports which step it is processing, It does not report the elapsed time of each step. When comparing results, test_models reports file counts and reports by name the first five differing or orphaned files.
This default behaviour can be modified by the following command options:
-
--allfilesReport each differing and orphaned file by name, not just the first five -
--timingAlso report the elapsed time for each step -
--nostepsDon't report which step is being processed
For example, on Windows, the command
perl test_models.pl -m ../models --nomodgen --nocomp --timing --allfiles --significant_digits 8 OzProj
instructs test_models to build and run just the OpenM++ version of OzProj, suppress the flavour comparison output, report elapsed time of each step, and report every differing file by name, not just the first five. For illustrative purposes, this invocation also changes the number of significant digits used to construct digests from 7 to 8 to deliberately produce differences between Current and Reference OzProj results. Output is similar to the following:
=========================
test_models 2.0
=========================
Testing: OzProj
ompp-win settings: compiler=omc.exe (2021-05-28 02:28 GMT) platform=x64 configuration=release
OzProj: ompp-win: Build and publish model and Default scenario
OzProj: ompp-win: Build time 0m 7s
warning => OzProj: ompp-win: 63 build warning(s) - see OzProj/test_models/current/ompp-win/logs/build.log
OzProj: ompp-win: Run model using OzProj.ini
OzProj: ompp-win: Run time 0m 2s
OzProj: ompp-win: Convert outputs (8 digits of precision)
OzProj: ompp-win: Convert time 0m 0s
OzProj: ompp-win: Create digests of current outputs
OzProj: ompp-win: Digest time 0m 0s
OzProj: ompp-win: Current vs. Reference: 13 the same (of 21)
DIFFERS ===> OzProj: ompp-win: Current vs. Reference: 8 differ
DIFFERS ===> OzProj: ompp-win: Current vs. Reference: DIFFERS: 0_SIGNIFICANT_DIGITS.txt
DIFFERS ===> OzProj: ompp-win: Current vs. Reference: DIFFERS: Experiment1.csv
DIFFERS ===> OzProj: ompp-win: Current vs. Reference: DIFFERS: Experiment2.csv
DIFFERS ===> OzProj: ompp-win: Current vs. Reference: DIFFERS: Experiment3.csv
DIFFERS ===> OzProj: ompp-win: Current vs. Reference: DIFFERS: Experiment4.csv
DIFFERS ===> OzProj: ompp-win: Current vs. Reference: DIFFERS: Experiment5.csv
DIFFERS ===> OzProj: ompp-win: Current vs. Reference: DIFFERS: Experiment6.csv
DIFFERS ===> OzProj: ompp-win: Current vs. Reference: DIFFERS: PersonYearsLived.csv
The specially-named 'result' file 0_SIGNIFICANT_DIGITS.txt is described elswhere in this topic.
[back to arguments and options]
[back to topic contents]
Unless instructed otherwise, test_models builds a Release version of a model, without MPI multi-processing capability, and does not save copy of generated C++ code.
This default behaviour can be modified by the following command options:
-
--configBuild configuration: debug or release(default) -
--mpi_processesBuild MPI version (default 0, means no MPI) -
--gencodeSave a copy of the generated C++ code in subfoldergenerated_code
The Debug version of a model is typically considerably slower than the Release version. The Debug version can, however, perform run-time checks which are absent in the Release version. This is particularly true if C++ model code uses assert↗ to verify that expected logical conditions are satisfied at run time (in Debug mode only).
Supply a non-zero value to --mpi_processes to build an MPI-enabled model (OpenM++ only). The value will be used to launch that number of MPI instances when the model is subsequently run as described in Run. The suffix _mpi will be appended to the name of the model executable to distinguish it from the normal non-MPI version.
The specialized --gencode option creates a copy of the C++ code generated by the omc compiler (or by the Modgen compiler for the modgen flavour). If this option is activated, the src temporary build folder is copied to the output folder generated_code. The --gencode option can be used to compare the C++ code generated by different versions of the OpenM++ compiler.
Incidentally, models are capable of multi-threading independent of MPI.
[back to arguments and options]
[back to topic contents]
Unless instructed otherwise, test_models builds and runs a non-MPI enabled model using the Default scenario. It retains all files from the build, including the model executable and database after the model is run. Also by default, test_models looks in the model root for a model ini file to use to run the model. The search for the ini file proceeds as follows: If the file test_models.ini is present, it is used. If test_models.ini is not present, the file MODEL.ini is used if present, where MODEL stands for the name of the model. For more on how test_models uses a model ini file, see Control - Files.
This default behaviour can be modified by the following command options:
-
--iniThe name of the OpenM++ model ini used when the model is run -
--mpi_processesThe number of mpi processes to launch -
--cleanRemove all build files after the run
For example, if the file OM_ROOT/RiskPaths/Test.ini exists with content
[OpenM]
SubValues = 16
Threads = 4
[Parameter]
SimulationCases = 16000000and the following command is issued
perl test_models.pl -m ../models --newref --nomodgen --nocomp --timing --mpi_processes 4 --ini Test.ini RiskPaths
an MPI-enabled version of RiskPaths is built and 4 instances are launched on the workstation under MPI control. The run consists of 16 replicates with 1,000,000 cases in each replicate, for a total of 16,000,000 cases. Each MPI instance will run with 4 threads.
Output is similar to the following:
=========================
test_models 2.0
=========================
Testing: RiskPaths
ompp-win settings: compiler=omc.exe (2021-05-28 02:28 GMT) platform=x64 configuration=release mpi_processes=4
RiskPaths: ompp-win: Deleting previous Reference information
RiskPaths: ompp-win: Build and publish model and Default scenario
RiskPaths: ompp-win: Build time 0m 5s
warning => RiskPaths: ompp-win: 10 build warning(s) - see RiskPaths/test_models/current/ompp-win/logs/build.log
RiskPaths: ompp-win: Run model using Test.ini
RiskPaths: ompp-win: Run time 2m 15s
RiskPaths: ompp-win: Convert outputs (7 digits of precision)
RiskPaths: ompp-win: Convert time 0m 0s
RiskPaths: ompp-win: Create digests of current outputs
RiskPaths: ompp-win: Digest time 0m 0s
RiskPaths: ompp-win: No Reference outputs - create using Current outputs
RiskPaths: ompp-win: Current vs. Reference: 9 the same (of 9)
The truncated log file of the run is similar to the following:
2021-05-31 16:18:57.819 RiskPaths
2021-05-31 16:18:57.819 RiskPaths
2021-05-31 16:18:57.993 Parallel run of 4 modeling processes, 4 thread(s) each
2021-05-31 16:18:57.994 Model build : Windows 64 bit Release
2021-05-31 16:18:57.995 Prepare fixed and missing parameters
2021-05-31 16:18:58.012 Run: 102 Default
2021-05-31 16:18:58.012 Get scenario parameters for process
2021-05-31 16:18:58.013 Model build : Windows 64 bit Release
2021-05-31 16:18:58.013 Prepare fixed and missing parameters
2021-05-31 16:18:58.013 Run: 102 Default
2021-05-31 16:18:58.014 Get scenario parameters for process
2021-05-31 16:18:58.014 member=0 Bind scenario parameters
2021-05-31 16:18:58.014 member=0 Compute derived parameters
2021-05-31 16:18:58.014 member=0 Prepare for simulation
2021-05-31 16:18:58.015 member=1 Bind scenario parameters
2021-05-31 16:18:58.015 member=1 Compute derived parameters
2021-05-31 16:18:58.015 member=1 Prepare for simulation
2021-05-31 16:18:58.015 member=0 Simulation progress=0% cases=0
...
[back to arguments and options]
[back to topic contents]
After running a model, test_models extracts each output table from the model database and saves a copy of the original data as well as a version rounded to 6 significant digits. test_models then computes and stores a digest of each rounded csv file which it uses for subsequent comparisons. When a rounded table differs, test_models verifies that at least one cell differs by more than one part per million using the original table data, before reporting the table as different.
This default behaviour can be modified by the following command options:
-
--significant_digitsThe number of significant digits for table result comparison -
--zero_fuzzSet table cell values to zero if within this value of zero -
--nocellsDisable fallback cell-by-cell verification of differing tables and copy of original data
The number of significant digits used for rounding can be modified using the --significant_digits option. The default value is 6 (one part in one million). This level of precision can help reduce the number of differing tables reported by test_models while still identifying differences of substantive or logical significance.
Rounding reduces the number of spurious differences which would otherwise be reported by test_models. It is not unusual for different model versions to produce slightly different results, often at the level of numerical precision (15-16 digits of precision). This can occur due to different C++ compiler optimizations, or by logically equivalent but slightly different algorithms in models. Rounding to less precision reduces the number of spurious differences reported by test_models.
Most spurious differences are eliminated by rounding, but not all. For example, two numbers can differ only at the 7th digit of precision but round to different values at 6 digits of precision if the two numbers happen to fall on different sides of a 'rounding boundary' (last digit 5). test_models eliminates these false positives by verifying differing rounded tables using the original unrounded data. In practice, this verification is rapid. This verification can be disabled using the --nocells command line option. The --nocells option also disables the creation of original (unrounded) copies of model output tables.
The value of a table cell can be undefined, eg the mean value of an attribute for a cell with no observations. This is not the same as a table cell value of zero, eg a mean value of 0.0 calculated from one or more observations. test_models considers an undefined value to be different from the value 0.
Table measures using subtraction can sometimes produce values very close to zero as an artifact of floating-point computations. The presence of such values can produce false positives: tables which test_models considers different but which are not different substantively. test_models uses a zero-fuzz value to handle this situation. If the absolute value of a table cell is within zero_fuzz of 0.0, the table cell will be set to 0.0. The default zero_fuzz value is 1e-15. The zero_fuzz value can be set to 0 to disable zero_fuzz altogether.
test_models computes digests using the MD5 cryptographic algorithm. For example, the MD5 digest of the RiskPaths table T04_FertilityRatesByAgeGroup in the Default run, rounded to 6 digits and normalized to csv is the hexadecimal value 993ead71da6eed5dde398342278f629a. The MD5 digest is always 32 hexadecimal digits in length, independent of the size of the input file. Files with different digests are guaranteed to be different, and it is extremely unlikely (read almost impossible) for different files to have the same digest.
[back to arguments and options]
[back to topic contents]
The following options are informational.
-
--helpreport usage message and exit -
--versionreport test_models version and exit
They have short form versions -h and -v.
If either option is on the command line, test_models will display the requested information and immediately exit.
[back to arguments and options]
[back to topic contents]
The following options affect the normalization of event trace output, if present..
-
--time_format STRtime format for event trace (default 13.6f) -
--modgen_id_offset INToffset adjustment to Modgen IDs (default 0)
The --time_format option changes the precision of time in the normalized event trace report. For example, specifying --time_format 19.12f increases the precision from 6 to 12 digits after the decimal point, producing output like
1969.960131010721 BirthdayEvent (Person 29)
1970.960131010721 timeBirthdayEvent (Person 29)
The --modgen_id_offset option corrects the entity ID in the Modgen normalized event trace report. This is helpful if the run reproduces a single case from a larger run using a case seed. For such runs, Modgen assigns entity IDs incorrectly, causing spurious differences with ompp event trace output. To fix the issue, the entity IDs in the Modgen event trace report need to be offset by the number of the sub they came from. That sub # may already be known, but if not it can be calculated from the case seed by dividing the case seed by 2^31 and taking the integer part of the result. For example, to determine the sub # of the case seed 11636581014, divide it by 2147483648 giving 5.4187052948.... The sub # of the case is 5. Specifying --modgen_id_offset 5 will offset all entity IDs in the Modgen event trace report by 5, correcting the error and bringing the Modgen event trace report into alignment.
[back to arguments and options]
[back to topic contents]
-
--windows_platformWindows platform: x64(default) or Win32 -
--modgen_platformModgen platform: Win32(default) or x64
If desired, a 32-bit OpenM++ version of a model can be built instead of a 64-bit version, on Windows only, using the --windows_platform option.
Possibly, a 64-bit version of a Modgen model can be built instead of a 32-bit version using the --modgen_platform option. The build or run may fail, and run functionality may be limited or compromised.
[back to arguments and options]
[back to topic contents]
=== under construction ===
test_model operation is controlled mainly by command-line options, but also by the presence and contents of optional specially-named files in the model folder.
OpenM++ model ini files are described
here.
test_models always enables
Model development options
by automatically specifying the -OpenM.IniAnyKey to the model when a model ini file is used.
=== under construction ===
test_models writes a report line by line to the command window where it was invoked as it carries out operations.
test_models produces two kinds of output: A
report
it writes to the command window where it was invoked, and files which record and persist information about the model runs it performs. The files created by test_folders can be useful to probe details behind the summary information in a test_models
report.
For example, test_models reports a count of warnings during model build and also creates the file build.log containing more information about those warnings.
This subtopic contains the following sections.
- test_models folder: The test_models folder hierarchy and contents
- table outputs: The tables from a model run
- event trace: The normalized event trace from a model run
The output files generated by test_models are in the subfolder test_models of the model folder. The test_models subfolder and hierarchy are created automatically by test_models as needed. The test_models subfolder can be deleted at any time, and will be recreated on a subsequent invocation of test_models.
The following diagram shows the hierarchical structure of the test_models folder for the RiskPaths model, with some other subfolders of RiskPaths shown for context. In this example, test_models was previously invoked for the three flavours ompp-win, ompp-linux, and modgen.
RiskPaths
├───code
├───modgen
│ ├───bin
│ └───src
├───ompp
│ ├───bin
│ └───src
├───ompp-linux
│ ├───bin
│ └───src
├───parameters
│ └───Default
└───test_models
├───current
│ ├───modgen
│ │ ├───logs
│ │ └───outputs
│ │ └───original
│ ├───ompp-linux
│ │ ├───logs
│ │ └───outputs
│ │ └───original
│ └───ompp-win
│ ├───logs
│ └───outputs
│ └───original
└───reference
├───modgen
│ ├───logs
│ └───outputs
│ └───original
├───ompp-linux
│ ├───logs
│ └───outputs
│ └───original
└───ompp-win
├───logs
└───outputs
└───original
test_models organizes information using two folder hierarchy levels, as illustrated. For example, RiskPaths results for the Reference run with the ompp-win flavour are in the folder RiskPaths/test_models/reference/ompp-win.
Below is a list of all files produced by test_models within each flavour folder, eg within RiskPaths/current/ompp-win:
-
tombstone.txt- Contains basic information about the compiler, members, ini file, etc. used to build and run the model.test_modelsreports if this file differs between Current and Reference. -
logs/build.log- A copy of captured console output from all stages of the build process. -
logs/run.log- A copy of the log file produced by the model when run. -
outputs/trace.txt- A normalized version of the event trace file, if produced by the model. -
outputs/0_MODEL_INI.txt- The contents of the model ini file used to run the model, if any. -
outputs/0_SIGNIFICANT_DIGITS.txt- Contains the number of significant digits used to round model output tables before computing digests. -
outputs/digests.txt- Contains the name and digest value of each file in the outputs folder (excluding subfolders). -
outputs/*.csv- Model output tables, rounded. -
outputs/original/*.csv- Model output tables, original (unrounded).
[back to output files]
[back to topic contents]
Each table is saved in a flat csv format with a header line. Zero-based indexes identify the table cell. Each row consists of all classificatory dimension indices, followed by the single measure dimension index, followed by the cell value. The order of classificatory dimensions is the same as the order in the table statement which declared the table in model code. Classificatory dimensions will include an additional trailing level for the margin, if a margin is specified for the dimension in the table declaration. The measure index is always present even if there is only one measure in the measure dimension. The cell value can be empty. Here's an example extract:
Dim0,Dim1,Dim2,Dim3,Value
0,0,0,0,258.453
0,0,0,1,1119.96
0,0,0,2,933.714
0,0,0,3,86.1511
0,0,1,0,172.302
...
[back to output files]
[back to topic contents]
=== under construction ===
Note that times >= 32767 will be shown as 99999, as will +inf, to improve comparability between ompp and Modgen event trace files.
Note options which control normalized event trace operation.
Ompp models must set EventTrace.Style to modgen for event trace files to be normalized and comparable.
[back to output files]
[back to topic contents]
This example illustrates how to arrange that the default test_models run is fast, and is easier to debug or investigate if run-time anomalies or unexpected outputs occur during model development. The underlying idea is to develop a model in small incremental steps, and use test_models to "fail fast, fail early", after each incremental change to the model. This approach is efficient only if the testing phase is fast. This example uses the OncoSimX model on the Windows platform.
Step 1 investigates how the model performs with a normal Default run. First, a fresh version of the model is created using git to clone from the git server to the temporary folder C:/Temp/OncoSimX. Next, test_models is run with the --timing option to evaluate the performance of the Default run.
C:\temp\OncoSimX>%OM_ROOT%\bin\test_models -m .. --nomodgen --nocomp --timing OncoSimX
=========================
test_models 2.1
=========================
Testing: OncoSimX
ompp-win settings: compiler=omc.exe (2021-06-29 06:31 GMT) platform=x64 configuration=release
OncoSimX: ompp-win: Build and publish model and Default scenario
OncoSimX: ompp-win: Build time 1m 47s
warning => OncoSimX: ompp-win: 1794 build warnings - see OncoSimX/test_models/current/ompp-win/logs/build.log
OncoSimX: ompp-win: Run model using OncoSimX.ini
OncoSimX: ompp-win: Run time 25m 3s
OncoSimX: ompp-win: Convert outputs (6 digits of precision)
OncoSimX: ompp-win: Convert time 1m 33s
OncoSimX: ompp-win: Create digests of current outputs
OncoSimX: ompp-win: Digest time 0m 0s
OncoSimX: ompp-win: No Reference outputs - create using Current outputs
OncoSimX: ompp-win: Current vs. Reference: 215 the same (of 215)
OncoSimX: ompp-win: Compare time 0m 0s
The test_models report shows that the run itself took 25m, and that the run used the model ini file OncoSimX.ini. An ini file with the same name as the model is used by default if it exists. Here's the contents of that file:
[OpenM]
SubValues = 12
Threads = 6
[Parameter]
SimulationCases = 1000000
SimulationSeed = 16807
; Complete example of ini-file located at: props/model/ompp/Model-example.iniThe run had 1,000,000 cases divided among 12 members (aka replicates or sub-samples), and was accelerated by running 6 threads in parallel (the workstation for the run had 8 processor cores). As a test, 1,000,000 cases does explore many conditions in the model and can produce statistically meaningful results, but a run of that size with OncoSimX is not suitable for quick iteration during model development. Moreover, a multi-threaded run is much more difficult to debug than a single-threaded run, making it a poor choice for testing during development. Ignoring run time, test_models took about 3m 15s. If the run were modified to take 1m 45s of run time instead, with one thread of execution, the total time would be about 5m, which would be acceptable for rapid testing.
Step 2 creates a new model ini file named test_models.ini based on the original model ini file OncoSimX.ini with several modifications. The number of members is changed to 1, the number of threads to 1, and the the number of cases changed somewhat arbitrarily to 5,000 to see how fast that will turn out:
[OpenM]
SubValues = 1
Threads = 1
[Parameter]
SimulationSeed = 1
SimulationCases = 5000The choice of file name for the modified model ini file was not arbitrary. test_models will use by default a model ini file named test_models.ini if it exists.
Step 3 runs test_models with the new model ini test_models.ini which it will use by default. The --newref option is used to replace the previous results to avoid reporting meaningless differences.
C:\temp\OncoSimX>%OM_ROOT%\bin\test_models -m .. --nomodgen --nocomp --timing --newref OncoSimX
=========================
test_models 2.1
=========================
Testing: OncoSimX
ompp-win settings: compiler=omc.exe (2021-06-29 06:31 GMT) platform=x64 configuration=release
OncoSimX: ompp-win: Deleting previous Reference information
OncoSimX: ompp-win: Build and publish model and Default scenario
OncoSimX: ompp-win: Build time 1m 46s
warning => OncoSimX: ompp-win: 1794 build warnings - see OncoSimX/test_models/current/ompp-win/logs/build.log
OncoSimX: ompp-win: Run model using test_models.ini
OncoSimX: ompp-win: Run time 1m 41s
OncoSimX: ompp-win: Convert outputs (6 digits of precision)
OncoSimX: ompp-win: Convert time 1m 19s
OncoSimX: ompp-win: Create digests of current outputs
OncoSimX: ompp-win: Digest time 0m 1s
OncoSimX: ompp-win: No Reference outputs - create using Current outputs
OncoSimX: ompp-win: Current vs. Reference: 215 the same (of 215)
OncoSimX: ompp-win: Compare time 0m 0s
The report shows that test_models.ini was used as intended. The total time for test_models to build, run, and compare results was 4m 47s, which is about what was targeted as a quick test for incremental model development.
Step 4 uses git to add test_models.ini to the local repository in C:/Temp/OncoSimX, and then push the change to the server repository to persist it and make it available to others on the modelling team.
Step 5 runs test_models a final time using test_models.ini to see how long the Modgen version takes.
C:\Development\X\ompp\Perl>perl test_models.pl -m ../../models --noompp --timing OncoSimX
=========================
test_models 2.1
=========================
Testing: OncoSimX
modgen settings: version=12,1,3,0 (2019-12-19 20:31 GMT) platform=Win32 configuration=release
OncoSimX: modgen: Build model and prepare Default scenario
OncoSimX: modgen: Build time 0m 15s
warning => OncoSimX: modgen: 1 build warnings - see OncoSimX/test_models/current/modgen/logs/build.log
OncoSimX: modgen: Run model using test_models.ini
OncoSimX: modgen: Run time 38m 54s
warning => OncoSimX: modgen: 1 run warnings - see OncoSimX/test_models/current/modgen/logs/run.log
OncoSimX: modgen: Convert outputs (6 digits of precision)
OncoSimX: modgen: Convert time 12m 3s
OncoSimX: modgen: Create digests of current outputs
OncoSimX: modgen: Digest time 0m 1s
OncoSimX: modgen: Current vs. Reference: 205 the same (of 205)
OncoSimX: modgen: Compare time 0m 0s
Even with the small run in test_models.ini, the Modgen version takes 50m 13s, making it not suitable for quick development tests for the OncoSimmX model with all tables. The Modgen version simulates quickly, but writing all table outputs and converting those outputs takes a long time. The reason is that Modgen uses a Microsoft Access database for storing tables, and MS Access is slow.
The folder C:/Temp/OncoSimX is now deleted, having served its purpose.
In this example, a new OpenM++ release has been announced, and a model developer is asked to verify that the team's model successfully builds and runs with the new version and produces identical results. The model developer uses a Windows workstation.
Step 1 downloads and extracts the new version of OpenM++, released on 2021-06-29, to a new folder on the workstation. To simplify changing between OpenM++ versions, the model developer previously created a folder named C:/ompp_versions on the workstation. A new command prompt is opened and used to verify the contents of that folder after the download and extraction:
C:\ompp_versions>dir
Volume in drive C is OS
Volume Serial Number is 14E2-D15F
Directory of C:\ompp_versions
2021-06-29 07:27 AM <DIR> .
2021-06-29 07:27 AM <DIR> ..
2021-06-12 09:49 PM <DIR> openmpp_win_20210505
2021-06-12 09:50 PM <DIR> openmpp_win_20210602
2021-06-12 09:50 PM <DIR> openmpp_win_20210612
2021-06-29 07:27 AM <DIR> openmpp_win_20210629
0 File(s) 0 bytes
6 Dir(s) 1,717,655,265,280 bytes free
A command is issued to verify the version of OpenM++ currently in use on the workstation:
C:\ompp_versions>echo %OM_ROOT%
C:\ompp_versions\openmpp_win_20210602
The version currently in use is 2021-06-02 which is two versions old because the development team decided to skip the mid-month OpenM++ bug-fix release because that bug was known not to affect the team's model.
Step 2 prepares the test. The OncoSimX model is used in this example and it's assumed that the model developer is working actively on the model and has unchecked code changes. To not interfere with that current work-in-progress, git is used to create a new temporary clone of the stable trunk version of OncoSimX in the temporary folder C:/Temp/OncoSimX.
Step 3 invokes test_models to create a Reference run using the team's working version (2021-06-02) of OpenM++:
C:\ompp_versions>cd C:\Temp\OncoSimX
C:\temp\OncoSimX>perl %OM_ROOT%/perl/test_models.pl -m .. --nomodgen --nocomp OncoSimX
=========================
test_models 2.0
=========================
Testing: OncoSimX
ompp-win settings: compiler=omc.exe (2021-06-02 03:23 GMT) platform=x64 configuration=release
OncoSimX: ompp-win: Build and publish model and Default scenario
warning => OncoSimX: ompp-win: 1794 build warning(s) - see OncoSimX/test_models/current/ompp-win/logs/build.log
OncoSimX: ompp-win: Run model using test_models.ini
OncoSimX: ompp-win: Convert outputs (7 digits of precision)
OncoSimX: ompp-win: Create digests of current outputs
OncoSimX: ompp-win: No Reference outputs - create using Current outputs
OncoSimX: ompp-win: Current vs. Reference: 215 the same (of 215)
The -m .. option specifies that the folder containing models is the parent folder of the current folder.
Step 4 switches to the new OpenM++ release dated 2021-06-29 by changing the OM_ROOT environment variable:
C:\temp\OncoSimX>set OM_ROOT=C:\ompp_versions\openmpp_win_20210629
and invokes test_models again using the same arguments:
C:\temp\OncoSimX>perl %OM_ROOT%/perl/test_models.pl -m .. --nomodgen --nocomp OncoSimX
=========================
test_models 2.1
=========================
Testing: OncoSimX
ompp-win settings: compiler=omc.exe (2021-06-29 06:31 GMT) platform=x64 configuration=release
OncoSimX: ompp-win: Build and publish model and Default scenario
warning => OncoSimX: ompp-win: 1794 build warnings - see OncoSimX/test_models/current/ompp-win/logs/build.log
OncoSimX: ompp-win: Run model using test_models.ini
OncoSimX: ompp-win: Convert outputs (6 digits of precision)
OncoSimX: ompp-win: Create digests of current outputs
OncoSimX: ompp-win: Current vs. Reference: tombstone info differs:
OncoSimX: ompp-win: Reference: compiler=omc.exe (2021-06-02 03:23 GMT) platform=x64 configuration=release members=1
OncoSimX: ompp-win: Current: compiler=omc.exe (2021-06-29 06:31 GMT) platform=x64 configuration=release members=1
OncoSimX: ompp-win: Current vs. Reference: 214 the same (of 215)
DIFFERS ===> OncoSimX: ompp-win: Current vs. Reference: 1 differ
DIFFERS ===> OncoSimX: ompp-win: Current vs. Reference: DIFFERS: 0_SIGNIFICANT_DIGITS.txt
test_models reports that the OpenM++ compiler versions for the Reference and Current runs differ, as expected. Unexpectedly, it reported a single differing file. The name of that file and the two test_models reports pinpoint the cause: The default number of digits used by test_models was changed from 7 digits to 6 digits between the two OpenM++ releases. The change in the number of significant digits turned out not to affect the rounded versions of the model output tables in the two runs.
The test, at this point, has established that the model builds and runs successfully with the new version of OpenM++ and produces identical outputs for the small fast run used normally by test_models during OncoSimX development.
In step 5, the model developer decides to perform a more demanding verification by comparing results between the two OpenM++ releases using a larger run of 1,000,000 cases and 12 replicates. These are the settings in the model ini file OncoSimX/OncoSimX.ini:
[OpenM]
SubValues = 12
Threads = 6
[Parameter]
SimulationCases = 1000000
SimulationSeed = 16807
; Complete example of ini-file located at: props/model/ompp/Model-example.iniThe OpenM++ version is switched back to the older 2021-06-02 version
C:\temp\OncoSimX>set OM_ROOT=C:\ompp_versions\openmpp_win_20210602
and test_models is invoked, this time explicitly specifying the number of significant digits as well as the model ini file to use. The --newref option is used to create a new Reference run using the larger run specified in the ini file.
C:\temp\OncoSimX>perl %OM_ROOT%/perl/test_models.pl -m .. --nomodgen --nocomp --significant_digits 6 --ini OncoSimX.ini --newref OncoSimX
=========================
test_models 2.0
=========================
Testing: OncoSimX
ompp-win settings: compiler=omc.exe (2021-06-02 03:23 GMT) platform=x64 configuration=release
OncoSimX: ompp-win: Deleting previous Reference information
OncoSimX: ompp-win: Build and publish model and Default scenario
warning => OncoSimX: ompp-win: 1794 build warning(s) - see OncoSimX/test_models/current/ompp-win/logs/build.log
OncoSimX: ompp-win: Run model using OncoSimX.ini
OncoSimX: ompp-win: Convert outputs (6 digits of precision)
OncoSimX: ompp-win: Create digests of current outputs
OncoSimX: ompp-win: No Reference outputs - create using Current outputs
OncoSimX: ompp-win: Current vs. Reference: 215 the same (of 215)
The OpenM++ version is switched once again to the new release 2021-06-29
C:\temp\OncoSimX>set OM_ROOT=C:\ompp_versions\openmpp_win_20210629
and test_models is run again to produce a comparable Current run and compare it to the Reference run:
C:\temp\OncoSimX>perl %OM_ROOT%/perl/test_models.pl -m .. --nomodgen --nocomp --significant_digits 6 --ini OncoSimX.ini OncoSimX
=========================
test_models 2.1
=========================
Testing: OncoSimX
ompp-win settings: compiler=omc.exe (2021-06-29 06:31 GMT) platform=x64 configuration=release
OncoSimX: ompp-win: Build and publish model and Default scenario
warning => OncoSimX: ompp-win: 1794 build warnings - see OncoSimX/test_models/current/ompp-win/logs/build.log
OncoSimX: ompp-win: Run model using OncoSimX.ini
OncoSimX: ompp-win: Convert outputs (6 digits of precision)
OncoSimX: ompp-win: Create digests of current outputs
OncoSimX: ompp-win: Current vs. Reference: tombstone info differs:
OncoSimX: ompp-win: Reference: compiler=omc.exe (2021-06-02 03:23 GMT) platform=x64 configuration=release members=12
OncoSimX: ompp-win: Current: compiler=omc.exe (2021-06-29 06:31 GMT) platform=x64 configuration=release members=12
OncoSimX: ompp-win: Current vs. Reference: 215 the same (of 215)
The larger run executed successfully with the new OpenM++ release, and all output results were the same to 6 significant digits.
In this example, the OM_ROOT environment variable was changed using the set command (not the setx command) to select an OpenM++ version. The changed value of OM_ROOT applied only to commands issued in the command prompt window used for the test. The value of OM_ROOT in other command prompt windows or Visual Studio sessions was unaffected by the test.
=== under construction ===
=== under construction ===
In this example, test_models is used to build and run Debug versions of a model, using both OpenM++ and Modgen versions. Debug versions run more slowly but contain more run-time checks which can identify errors, so it is a good idea to test the Debug versions from time to time. test_models1 makes that easy. This example uses the OncoSimX` model running on a Windows system.
In step 1, a fresh set of Reference runs are created:
The Modgen run took considerably longer than the OpenM++ run.
In this example, test_models is used to evaluate the performance difference between two versions of the same model, holding other effects constant. Specifically, this example seeks to measure the performance cost of the run-time checks which OpenM++ performs to ensure that attributes are not modified directly or indirectly in event time functions. Although those runtime checks are core OpenM++ functionality, the OpenM++ option verify_attribute_modification can be used to remove them.
Step 1 sets up the experiment. The OncoSimX model is chosen because it is a large-scale production model with many events. The experiment is done on a secondary (older) Windows workstation which will run no other tasks during the experiment. The experiment is temporary, so a single git command is issued to create a new clone of OncoSimX in a convenient temporary location, which in this example is C:/Temp/OncoSimX. The OncoSimX clone contains no test_models results, which is appropriate for a model git repository. It does contain the model ini file test_models.ini in the model root.
Step 2 sets up the test_models model run to serve as a performance test. Normally, a model run for test_models is small and fast. This experiment requires a run which is large enough to produce reliable performance information. The OncoSimX model ini file test_models.ini produces small fast runs for rapid development. The contents are:
[OpenM]
SubValues = 1
Threads = 1
[Parameter]
SimulationSeed = 1
SimulationCases = 5000
[Tables]
;Retain = \
; TG01_Breast_Cancer_TablesA single line is modified to increase the number of cases from 5,000 to 1,000,000 for the experiment.
SimulationCases = 1000000Step 3 uses test_models to create a Reference run to serve as a basis for comparison. A command window is opened and test_models is invoked:
perl test_models.pl -m C:/Temp --timing --nomodgen OncoSimX
The temporary OncoSimX clone was created in the folder C:/Temp, so that folder is specified as the models folder using the -m option. Timing information is requested using the --timing option. The experiment is being done on a Windows workstation with Modgen installed, so the --nomodgen argument is used to suppress that flavour.
The output is
=========================
test_models 2.1
=========================
Testing: OncoSimX
ompp-win settings: compiler=omc.exe (2021-07-01 12:27 GMT) platform=x64 configuration=release
OncoSimX: ompp-win: Build and publish model and Default scenario
OncoSimX: ompp-win: Build time 3m 58s
warning => OncoSimX: ompp-win: 1794 build warnings - see OncoSimX/test_models/current/ompp-win/logs/build.log
OncoSimX: ompp-win: Run model using test_models.ini
OncoSimX: ompp-win: Run time 98m 8s
OncoSimX: ompp-win: Convert outputs (6 digits of precision)
OncoSimX: ompp-win: Convert time 2m 10s
OncoSimX: ompp-win: Create digests of current outputs
OncoSimX: ompp-win: Digest time 0m 1s
OncoSimX: ompp-win: No Reference outputs - create using Current outputs
OncoSimX: ompp-win: Current vs. Reference: 215 the same (of 215)
OncoSimX: ompp-win: Compare time 0m 0s
OncoSimX: Flavour compare time 0m 0s
Note that test_models created Reference results from Current results because no Reference run was present. The reported run time was 98m 8s.
Step 4 modifies the model to remove the normal run-time checks for the experiment. Because this is a temporary clone of OncoSimX which will be discarded after the experiment, the source code can be modified freely with no consequences. The small source file TraceOptions.mpp was selected arbitrarily to implement the experiment, and the following line added:
options verify_attribute_modification = off;Step 5 uses test_models to create a Current run which will be compared to the Reference run. The invocation of test_models is identical to the previous invocation.
perl test_models.pl -m C:/Temp --timing --nomodgen OncoSimX
The output is
=========================
test_models 2.1
=========================
Testing: OncoSimX
ompp-win settings: compiler=omc.exe (2021-07-01 12:27 GMT) platform=x64 configuration=release
OncoSimX: ompp-win: Build and publish model and Default scenario
OncoSimX: ompp-win: Build time 3m 58s
warning => OncoSimX: ompp-win: 1794 build warnings - see OncoSimX/test_models/current/ompp-win/logs/build.log
OncoSimX: ompp-win: Run model using test_models.ini
OncoSimX: ompp-win: Run time 95m 16s
warning => OncoSimX: ompp-win: 1 run warnings - see OncoSimX/test_models/current/ompp-win/logs/run.log
OncoSimX: ompp-win: Convert outputs (6 digits of precision)
OncoSimX: ompp-win: Convert time 2m 12s
OncoSimX: ompp-win: Create digests of current outputs
OncoSimX: ompp-win: Digest time 0m 0s
OncoSimX: ompp-win: Current vs. Reference: 215 the same (of 215)
OncoSimX: ompp-win: Compare time 0m 0s
OncoSimX: Flavour compare time 0m 0s
All model outputs are identical as expected. test_models reports one run-time warning which was absent in the first run. That's because OpenM++, by design, emits a warning on every run of a model which disables run-time checking of event times. test_models noted the warning and reported it when it analyzed the run log. The reported run time was 95m 16s.
Step 6 compares performance between the Reference and Current runs. The model run time in the Current run was 172s less than the Reference run, and shows that about 2.9% of the elapsed time of the Reference run was overhead associated with the run-time checks. That is a realistic assessment of the overhead of the run-time checks for OncoSimX when used in production.
The elapsed time reported by test_models includes the time required to read parameters and to assemble and write tables, which can be significant. It might also be interesting to assess the performance effect on the simulation time alone, removing the fixed cost of parameter reading and table assembly and writing. The model log files saved by test_models can be used to perform that calculation.
Here's a line extracted from the Reference log file, located at
C:/Temp/OncoSimX/test_models/reference/ompp-win/logs/run.log:
2021-07-01 10:16:52.928 member=0 Simulation summary: cases=1000000, events/case=1128.9, entities/case=5.4, elapsed=5693.747633s
and here's the corresponding line extracted from the Current log file, located at
C:/Temp/OncoSimX/test_models/current/ompp-win/logs/run.log.
2021-07-01 12:31:45.777 member=0 Simulation summary: cases=1000000, events/case=1128.9, entities/case=5.4, elapsed=5525.430919s
Conveniently, this line in the OpenM++ model log reports the elapsed time of the computational phase, in seconds. The simulation time was 5694s for the Reference run and 5525s for the Current run, for a difference of 168s. So about 3.0% of the computation time was overhead associated with the run-time checks. That's only slightly higher than the proportion computed using elapsed time because the time required to read parameters and to assemble and write tables (about 190s) is a relatively small portion of the elapsed time of an OncoSimX run of 1,000,000 (5525s).
=== under construction ===
The following applies to a case-based model.
If test_models detects unexpected output differences which are not immediately explained by recent model code changes or by perusing differences in specific tables, it can be useful to determine whether the differences are due to a handful of different cases in the run, or to some other cause.
To find out, turn on the case checksum option by adding the statement
options case_checksum = on;
to model code. In the RiskPaths example model, this is done in a dedicated model source file named TraceOptions.mpp whcih is normally:
//LABEL(TraceOptions, EN) Control options for trace output
/* NOTE(TraceOptions, EN)
The options must be set to off for normal use.
If the options are on, very large trace output files may be produced by the model.
*/
options event_trace = off;
options case_checksum = off;Turning case_checksum on will cause a trace file to be generated when test_models runs the model which contains one line for each case. For RiskPaths, the first lines of the trace file look like this:
Seed: 1 Sample: 0 Checksum: 4.4094661700e+02
Seed: 470583131 Sample: 0 Checksum: 4.1293225500e+02
Seed: 1715377866 Sample: 0 Checksum: 3.4860055300e+02
Seed: 430166727 Sample: 0 Checksum: 1.3482143400e+02
Seed: 743781969 Sample: 0 Checksum: 1.4831547800e+02
Seed: 594673803 Sample: 0 Checksum: 4.1803896100e+02
Seed: 2048386738 Sample: 0 Checksum: 3.5386231900e+02
Seed: 1681919254 Sample: 0 Checksum: 4.0119902000e+02
Seed: 78320563 Sample: 0 Checksum: 3.9439706500e+02
Seed: 1637697257 Sample: 0 Checksum: 4.9065368400e+02
Seed: 127732046 Sample: 0 Checksum: 9.8715092900e+02
Seed: 2114816392 Sample: 0 Checksum: 3.9576701300e+02
Seed: 1616242686 Sample: 0 Checksum: 1.2478963010e+03
Seed: 1222943222 Sample: 0 Checksum: 4.5861339400e+02
Seed: 170881636 Sample: 0 Checksum: 1.4320257000e+02
This file is noticed by test_models and will participate in test_models comparisons. Test_models saves the trace file under the name trace.txt in the same folder where it places csv versions of tables.
If test_models reports that the file trace.txt differs between the ompp and modgen versions, then one or more cases differ in the two flavours. Use a tool or text editor to compare the two versions of trace.txt to determine the seed of the first case which has a differing checksum in the two variants.
More to follow...