test models - openmpp/openmpp.github.io GitHub Wiki
Home > Model Development Topics > Test Models
This topic contains detailed information about the OpenM++ test_models
utility. test_models
builds, runs, and compares results for different versions of the same model, or for the same model built in different ways or on different platforms.
- Introduction and overview
- Windows Quick start
- Linux or MacOS Quick start
- Concepts
- Arguments and Options
- Control - Files
- Output - Report
- Output - Files
- Example 1: Preparing the default run
- Example 2: A new OpenM++ release
- Example 3: A model code change (under construction)
- Example 4: When results differ (under construction)
- Example 5: Exercising a model in Debug (under construction)
- Example 6: A performance comparison
- Example 7: Using event trace output (under construction, some content present)
The test_models
utility builds, runs, and compares results for different versions of the same model, or for the same model built on different platforms. It can process multiple models in a single invocation and can compare OpenM++ and Modgen versions of cross-compatible models. It can play a useful role during incremental model development, and for quality assurance.
test_models
is efficient enough that it can be used routinely in model development workflow cycles, even for large models (using a small sized Default simulation). even multiple times per hour or more during active model development.
When used routinely to assess the effects of incremental model changes, test_models
can help identify bugs in model logic. It highlights when model changes have unexpected effects in model outputs, by comparing all cells of all output tables between a Current run and a Reference run.
test_models
can be used to compare outputs of ompp model versions across platforms. Such differences are rare, and are typically caused by errors in model logic (often due to bad C++ code in models, eg use of uninitialized variables, or accessing memory beyond array limits using index variables with bad values).
test_models
works with the standard model folder structure illustrated in the sample models in the OpenM++ distribution. It organizes results herarchically in a subfolder test_models
of the model folder. test_models
organizes comparisons in two dimensions. The first dimension is {Reference, Current}. A user can tell test_models
to move Current results to Reference results to create a new point of reference for subsequent output comparisons. The second dimension of comparison is the platform ‘flavour’ {modgen, ompp-win, ompp-linux, ompp-macos}. Arguments to test_models
determine where model results will be stored and which comparisons will be performed after test_models
does the requested runs. For speed and ease of use, each model output table is converted to three forms: full precision, rounded precision, and a summary 'digest' of the file contents. The use of rounded precision eliminates most (but not all) spurious differences, eg differences at 6-7th decimal place. The table digest makes it possible for test_models
to determine whether two versions of a table differ at very high speed, without comparing the two tables cell-by-cell.
test_models
can compare results of Modgen and ompp versions of models, despite large underlying differences in output storage schema and technology. It reads the database of each and converts output tables into a common comparable csv format.
test_models
can be very useful for cross-compatible model conversion. One can break cross-compatible model conversion into atomic steps and use test_models
after each atomic step to verify that Modgen results have not changed, even before an OpenM++ version of the model is buildable. The comparisons relevant during cross-compatible model conversion are
- Current(modgen) vs. Reference(modgen) - Modgen results should remain identical at each atomic change to model source, when build in Modgen.
- Current(ompp) vs. Current(modgen) – After the model code is cross-compatible, ie when the ompp version can be built with no compiler errors, the model can be run in both versions and model results should be identical, because the model specifications are identical (same model source code and same Default scenario). Sometimes they are not, in which case the tables which differ can sometimes provide a clue.
When tracking down the cause of an unexpected difference in two runs, it can cometimes be useful to simplify the Default scenario and run test_models
again, to see which area of the model might responsible for the difference. For example, if all tables differ, but when one turns off immigration in the Default scenario all differences disappear, one knows that the immigration code is somehow responsible. It can also be useful to turn on event tracking in the model with a small simulated population. If test_models
indicates that the two event tracking files are identical, but table results are different, then the cause could be a Modgen tabulation bug.
test_models
is also useful for model development. One can break model development into small atomic steps to help identify unexpected changes in model outputs. If possible, one can structure model changes so that Default parameters and code changes should have no effect on outputs, eg by turning a new model option off in Default parameters. That can identify whether the new model code, when disabled using the new parameters, has unexpected effects on other parts of the model (it should not). The test_models
comparisons typically used for model development are
- Current(ompp) vs. Reference(ompp) - Model results from incremental changes should affect only new tables or expected interactions among model components. Once verified, one can tell test_models to copy Current results to Reference results (using the
--newref
option) to create a new Reference for the next atomic set of changes to model code.
If one prefers to work in the Modgen environment for model development, one can instead use
- Current(modgen) vs. Reference(modgen) - See notes above.
- Current(modgen) vs. Current(ompp) – From time to time, build the ompp version to identify if non cross-compatible code has crept into the model source, and rectify if so.
test_models
also standardizes the detailed event trace outputs (if activated) in ompp and Modgen models so that they are comparable and readable. This can identify precisely when a simulation diverges in the Modgen and ompp versions of a model, or in (for example) two ompp versions of a model. This is useful for unexpected and hard-to-understand differences in simulations. If the time and event of earliest divergence in the two runs is insufficient to understand the cause in and of itself, that information provides what’s required to set conditional breakpoints for a parallel debugging session of the two runs, stepping through the simulation in each of the two debugging sessions, to identify the precise code location responsible for the divergence in the simulation in the two versions.
test_models
also notes some tombstone information about each set of outputs, eg the modgen version. It also keeps a copy of the build log output and Default run log output. test_models
can also report the elpased time of various steps, including model build and model run. A count of compiler warnings is also reported.
test_models
can run and process a single model, or multiple models in subfolders of a parent folder. That can be useful for testing mutliple scenarios of a single model, using mutliple git clones inside a parent folder. It can also be useful to bulk test multiple models routinely, for example after a change in the OpenM++ version.
A 64-bit Windows executable version of test_models
is distributed with OpenM++ at OM_ROOT/bin/test_models.exe
, where OM_ROOT
stands for the OpenM++ installation directory. A 32-bit version is at OM_ROOT/bin/test_models32.exe
. The 32-bit version may be required for the Modgen flavour to work successfully using the executable version of test_models
.
To test installation and operation of test_models
, open a command prompt, change the current directory to OM_ROOT/bin
, and type the command
test_models -v
Output should be similar to the following:
test_models version 2.1
test_models
is written in the Perl language, and distributed with OpenM++ at OM_ROOT/Perl/test_models.pl
. Most examples in this topic invoke test_models
using the Perl interpreter from the OM_ROOT/Perl
directory, eg
perl test_models.pl -v
On Windows, unless you have Perl and the required Perl components installed, invoke the executable version of test_models
from the OM_ROOT/bin
directory with a command like
test_models -v
or
test_models32 -v
From the OM_ROOT/bin
directory, type the command
test_models -h
Output should be similar to the following:
test_models [-hmv] [long options...] model...
-m STR --models_root STR directory containing models (default is .)
--newref replace Reference results with Current
results
--noompp skip OpenM++ build and run
--nomodgen skip Modgen build and run
--nocomp skip flavour comparison
--allfiles report all different and orphaned files
--timing report elapsed time of steps
--nosteps skip reporting which step is being
performed
--config STR build configuration: debug or
release(default)
--mpi_processes INT build MPI version and run with n
processes (default 0, means no MPI)
--gencode keep a copy of the generated C++ code
--ini STR OpenM++ model ini file to pass to model
(in model root, default is
test_models.ini if present)
--clean remove all build files after run
--significant_digits INT significant digits (default 6)
--nocells disable fallback cell-by-cell
verification of differing tables and copy
of original data
-h --help report usage message and exit
-v --version report test_models version and exit
--windows_platform STR Windows platform: x64(default) or Win32
--modgen_platform STR Modgen platform: Win32(default) or x64on of unrounded versions
of tables
From the OM_ROOT/bin
directory, type the command
test_models -m ../models RiskPaths
If Modgen is installed, use test_models32
instead of test_models
. test_models
will build and run both the OpenM++ and the Modgen versions of RiskPaths, and compare their results. Output should be similar to the following:
=========================
test_models 2.0
=========================
Testing: RiskPaths
modgen settings: version=12,1,3,0 (2019-12-19 20:31 GMT) platform=Win32 configuration=release
ompp-win settings: compiler=omc.exe (2021-05-28 02:28 GMT) platform=x64 configuration=release
RiskPaths: modgen: Build model and prepare Default scenario
RiskPaths: modgen: Run model using RiskPaths.ini
RiskPaths: modgen: Convert outputs (7 digits of precision)
RiskPaths: modgen: Create digests of current outputs
RiskPaths: modgen: No Reference outputs - create using Current outputs
RiskPaths: modgen: Current vs. Reference: 9 the same (of 9)
RiskPaths:
RiskPaths: ompp-win: Build and publish model and Default scenario
warning => RiskPaths: ompp-win: 10 build warning(s) - see RiskPaths/test_models/current/ompp-win/logs/build.log
RiskPaths: ompp-win: Run model using RiskPaths.ini
RiskPaths: ompp-win: Convert outputs (7 digits of precision)
RiskPaths: ompp-win: Create digests of current outputs
RiskPaths: ompp-win: No Reference outputs - create using Current outputs
RiskPaths: ompp-win: Current vs. Reference: 9 the same (of 9)
RiskPaths:
RiskPaths: Flavour comparisons:
RiskPaths:
RiskPaths: ompp-win vs. modgen: Reference: 9 the same (of 9)
RiskPaths:
RiskPaths: ompp-win vs. modgen: Current: 9 the same (of 9)
If Modgen is not installed, output will not include the Modgen portion nor the flavour comparison portions of the report.
test_models
uses the Microsoft application msbuild.exe
to build models. msbuild.exe
is normally installed as part of Visual Studio installation, but the install location can vary. test_models
attempts to determine the location of msbuild.exe
but may not always succeed. If you encounter issues running test_models
which seem related to msbuild.exe
you can provide test_models
the location explicitly using the environment variable MSBUILD_EXE
, for example:
set MSBUILD_EXE=C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Current\Bin\msbuild.exe
in the command window before invoking test_models
.
test_models
is a Perl script distributed with OpenM++ at OM_ROOT/Perl/test_models.pl
, where OM_ROOT
stands for the OpenM++ installation directory.
To test installation and operation of test_models
, open a command prompt, change the current directory to OM_ROOT\Perl
, and type the command
perl test_models.pl -v
Output should be similar to the following:
test_models version 2.1
Depending on your operating system version and installation history, Perl may ask you to install missing Perl modules required by test_models.pl
.
If so, it will name them explicitly when you invoke test_models.pl
.
We do recommend to use cpanm
for Perl modules installation. Typical scenario is:
cpan App::cpanminus # initialize cpanm, if not done before
cpanm Getopt::Long::Descriptive
cpanm Capture::Tiny
cpanm Config::Tiny
cpanm File::Copy::Recursive
cpanm File::Which
Above list of modules can be different and depends on your current Perl configuration, and on the version of test_models.
From the OM_ROOT/Perl
directory, type the command
perl test_models.pl -h
Output should be similar to the following:
test_models [-hmv] [long options...] model...
-m STR --models_root STR directory containing models (default is .)
--newref replace Reference results with Current
results
--noompp skip OpenM++ build and run
--nomodgen skip Modgen build and run
--nocomp skip flavour comparison
--allfiles report all different and orphaned files
--timing report elapsed time of steps
--nosteps skip reporting which step is being
performed
--config STR build configuration: debug or
release(default)
--mpi_processes INT build MPI version and run with n
processes (default 0, means no MPI)
--gencode keep a copy of the generated C++ code
--ini STR OpenM++ model ini file to pass to model
(in model root, default is
test_models.ini if present)
--clean remove all build files after run
--significant_digits INT significant digits (default 6)
--nocells disable fallback cell-by-cell
verification of differing tables and copy
of original data
-h --help report usage message and exit
-v --version report test_models version and exit
--windows_platform STR Windows platform: x64(default) or Win32
--modgen_platform STR Modgen platform: Win32(default) or x64
From the OM_ROOT/Perl
directory, type the command
perl test_models.pl -m ../models RiskPaths
Output should be similar to the following:
=========================
test_models 2.0
=========================
Testing: RiskPaths
ompp-linux settings: compiler=omc (2021-05-29 16:47 GMT) configuration=release
RiskPaths: ompp-linux: Build and publish model and Default scenario
RiskPaths: ompp-linux: Run model using RiskPaths.ini
RiskPaths: ompp-linux: Convert outputs (7 digits of precision)
RiskPaths: ompp-linux: Create digests of current outputs
RiskPaths: ompp-linux: No Reference outputs - create using Current outputs.
RiskPaths: ompp-linux: Current vs. Reference: 9 the same (of 9)
This subtopic describes key underlying concepts used in test_models
.
- Run version: Current or Reference
- Run flavour: Windows, Linux, MacOS, Modgen
- Output comparison:
- Build options:
- Run options:
=== under construction ===
- Current or Reference
[back to concepts]
[back to topic contents]
=== under construction ===
- Windows, Linux, MacOS, Modgen
[back to concepts]
[back to topic contents]
=== under construction ===
[back to concepts]
[back to topic contents]
=== under construction ===
[back to concepts]
[back to topic contents]
=== under construction ===
[back to concepts]
[back to topic contents]
This subtopic describes the command line options and arguments of test_models
, organized into sections. It also describes default test_models
behaviour for each section.
- Syntax: The syntax of test_models arguments
- Models: The models to process
- Actions: The actions to perform on a model
- Verbosity: The level of report detail
- Build: How a model is built
- Run: How a model is run
- Comparison: How result are compared
- Informational: Informational options
- EventTrace: EventTrace options
A complete list of options is displayed by issuing the command
perl test_models.pl -h
test_models [-hmv] [long options...] model...
-m STR --models_root STR directory containing models (default is .)
--newref replace Reference results with Current
results
--noompp skip OpenM++ build and run
--nomodgen skip Modgen build and run
--nocomp skip flavour comparison
--allfiles report all different and orphaned files
--timing report elapsed time of steps
--nosteps skip reporting which step is being
performed
--config STR build configuration: debug or
release(default)
--mpi_processes INT build MPI version and run with n
processes (default 0, means no MPI)
--gencode keep a copy of the generated C++ code
--ini STR OpenM++ model ini file to pass to model
(in model root, default is
test_models.ini if present)
--clean remove all build files after run
--significant_digits INT significant digits (default 6)
--zero_fuzz NUM zero fuzz value (default 1e-15)
--nocells disable fallback cell-by-cell
verification of differing tables and copy
of original data
-h --help report usage message and exit
-v --version report test_models version and exit
--time_format STR time format for event trace (default
13.6f)
--modgen_id_offset INT offset adjustment to Modgen IDs (default
0)
--windows_platform STR Windows platform: x64(default) or Win32
--modgen_platform STR Modgen platform: Win32(default) or x64
[back to arguments and options]
[back to topic contents]
Arguments to test_models
consist of a series of options followed by a list of models to process. Either or both may be empty, in which case default values are used. An option starts with --
followed immediately by the option name. Some options may be followed by an option value. Some options have a synonymous short form consisting of a single -
followed imediately by a single letter.
[back to arguments and options]
[back to topic contents]
The --models_root
option (short form -m
) specifies the parent folder of one or more folders each of which contains a model. If not specified, the models root is the current working directory. Consider the following command, issued after setting the current working directory to OM_ROOT/Perl
:
perl test_models.pl -m ../models NewCaseBased RiskPaths
On Windows without Perl installed, a similar command would be issued after setting the current working directory to OM_ROOT/bin
:
test_models -m ../models NewCaseBased RiskPaths
The command instructs test_models
to set the models root to the sister directory models
in the OpenM++ distribution. That folder is OM_ROOT/models
and contains several sample models distributed with OpenM++, each in its own subfolder, eg OM_ROOT/models/RiskPaths
. The final two arguments instruct test_models
to process the two models in the subfolders OM_ROOT/models/NewCaseBased
and OM_ROOT/models/RiskPaths
.
On Linux, output is similar to:
=========================
test_models 2.0
=========================
Testing: NewCaseBased, RiskPaths
ompp-linux settings: compiler=omc (2021-05-29 16:47 GMT) configuration=release
NewCaseBased: ompp-linux: Build and publish model and Default scenario
NewCaseBased: ompp-linux: Run model using test_models.ini
NewCaseBased: ompp-linux: Convert outputs (7 digits of precision)
NewCaseBased: ompp-linux: Create digests of current outputs
NewCaseBased: ompp-linux: No Reference outputs - create using Current outputs
NewCaseBased: ompp-linux: Current vs. Reference: 3 the same (of 3)
RiskPaths: ompp-linux: Build and publish model and Default scenario
RiskPaths: ompp-linux: Run model using RiskPaths.ini
RiskPaths: ompp-linux: Convert outputs (7 digits of precision)
RiskPaths: ompp-linux: Create digests of current outputs
RiskPaths: ompp-linux: No Reference outputs - create using Current outputs
RiskPaths: ompp-linux: Current vs. Reference: 9 the same (of 9)
On Windows with Modgen installed, output is similar to:
=========================
test_models 2.0
=========================
Testing: NewCaseBased, RiskPaths
modgen settings: version=12,1,3,0 (2019-12-19 20:31 GMT) platform=Win32 configuration=release
ompp-win settings: compiler=omc.exe (2021-05-28 02:28 GMT) platform=x64 configuration=release
NewCaseBased: modgen: Build model and prepare Default scenario
NewCaseBased: modgen: Run model using test_models.ini
NewCaseBased: modgen: Convert outputs (7 digits of precision)
NewCaseBased: modgen: Create digests of current outputs
NewCaseBased: modgen: No Reference outputs - create using Current outputs
NewCaseBased: modgen: Current vs. Reference: 3 the same (of 3)
NewCaseBased:
NewCaseBased: ompp-win: Build and publish model and Default scenario
warning => NewCaseBased: ompp-win: 3 build warning(s) - see NewCaseBased/test_models/current/ompp-win/logs/build.log
NewCaseBased: ompp-win: Run model using test_models.ini
NewCaseBased: ompp-win: Convert outputs (7 digits of precision)
NewCaseBased: ompp-win: Create digests of current outputs
NewCaseBased: ompp-win: No Reference outputs - create using Current outputs
NewCaseBased: ompp-win: Current vs. Reference: 3 the same (of 3)
NewCaseBased:
NewCaseBased: Flavour comparisons:
NewCaseBased:
NewCaseBased: ompp-win vs. modgen: Reference: 3 the same (of 3)
NewCaseBased:
NewCaseBased: ompp-win vs. modgen: Current: 3 the same (of 3)
RiskPaths: modgen: Build model and prepare Default scenario
RiskPaths: modgen: Run model using RiskPaths.ini
RiskPaths: modgen: Convert outputs (7 digits of precision)
RiskPaths: modgen: Create digests of current outputs
RiskPaths: modgen: No Reference outputs - create using Current outputs
RiskPaths: modgen: Current vs. Reference: 9 the same (of 9)
RiskPaths:
RiskPaths: ompp-win: Build and publish model and Default scenario
warning => RiskPaths: ompp-win: 10 build warning(s) - see RiskPaths/test_models/current/ompp-win/logs/build.log
RiskPaths: ompp-win: Run model using RiskPaths.ini
RiskPaths: ompp-win: Convert outputs (7 digits of precision)
RiskPaths: ompp-win: Create digests of current outputs
RiskPaths: ompp-win: No Reference outputs - create using Current outputs
RiskPaths: ompp-win: Current vs. Reference: 9 the same (of 9)
RiskPaths:
RiskPaths: Flavour comparisons:
RiskPaths:
RiskPaths: ompp-win vs. modgen: Reference: 9 the same (of 9)
RiskPaths:
RiskPaths: ompp-win vs. modgen: Current: 9 the same (of 9)
If no models are specified on the command line, test_models
will process All subfolders of the models root.
[back to arguments and options]
[back to topic contents]
Unless instructed otherwise, test_models
builds and runs all available 'flavours' of a model on the current operating system. On Windows, two flavours are available: OpenM++ and Modgen (if installed). On Linux and MacOS, the only available flavour is OpenM++. After building and running each flavour, test_models
compares Current results to existing Reference results. If there are no Reference results, test_models
creates them by copying Current results. After completing all flavours for a model, test_models
compares results for other flavours (if present) to results for the OpenM++ flavour on the current operating system.
This default behaviour can be modified by the following command options:
-
--newref
For each flavour in this invocation, discard all Reference results and replace them with Current results -
--noompp
Suppress processing the OpenM++ flavour -
--nomodgen
Suppress processing the Modgen flavour (only applies if invoked on Windows) -
--nocomp
Suppress reporting on differences between flavours
For example, on Windows, the command
test_models -m ../models --noompp --nomodgen RiskPaths
would process the RiskPaths model, but skip build and run of both Windows flavours (OpenM++ and Modgen). test_models
would still compare results for any previously run flavours to OpenM++ Windows results (if present). Output might be similar to the following (or could be empty, if no other flavours were run previously):
=========================
test_models 2.0
=========================
Testing: RiskPaths
RiskPaths:
RiskPaths: Flavour comparisons:
RiskPaths:
RiskPaths: ompp-win vs. modgen: Reference: 9 the same (of 9)
RiskPaths:
RiskPaths: ompp-win vs. modgen: Current: 9 the same (of 9)
RiskPaths:
RiskPaths: ompp-win vs. ompp-linux: Reference: 9 the same (of 9)
RiskPaths:
RiskPaths: ompp-win vs. ompp-linux: Current: 9 the same (of 9)
In this invocation, the RiskPaths model was neither built nor run. test_models
detected the presence of Current and Reference results for two other flavours (Modgen and OpenM++ on Linux), and compared them to results from OpenM++ on Windows which were present from a previous invocation of test_models
. Incidentally, because test_models
uses digests to compare results, comparing all results between two flavours is almost instantaneous, even for models with many tables or with very large tables.
[back to arguments and options]
[back to topic contents]
Unless instructed otherwise, test_models
reports which step it is processing, It does not report the elapsed time of each step. When comparing results, test_models
reports file counts and reports by name the first five differing or orphaned files.
This default behaviour can be modified by the following command options:
-
--allfiles
Report each differing and orphaned file by name, not just the first five -
--timing
Also report the elapsed time for each step -
--nosteps
Don't report which step is being processed
For example, on Windows, the command
perl test_models.pl -m ../models --nomodgen --nocomp --timing --allfiles --significant_digits 8 OzProj
instructs test_models
to build and run just the OpenM++ version of OzProj, suppress the flavour comparison output, report elapsed time of each step, and report every differing file by name, not just the first five. For illustrative purposes, this invocation also changes the number of significant digits used to construct digests from 7 to 8 to deliberately produce differences between Current and Reference OzProj results. Output is similar to the following:
=========================
test_models 2.0
=========================
Testing: OzProj
ompp-win settings: compiler=omc.exe (2021-05-28 02:28 GMT) platform=x64 configuration=release
OzProj: ompp-win: Build and publish model and Default scenario
OzProj: ompp-win: Build time 0m 7s
warning => OzProj: ompp-win: 63 build warning(s) - see OzProj/test_models/current/ompp-win/logs/build.log
OzProj: ompp-win: Run model using OzProj.ini
OzProj: ompp-win: Run time 0m 2s
OzProj: ompp-win: Convert outputs (8 digits of precision)
OzProj: ompp-win: Convert time 0m 0s
OzProj: ompp-win: Create digests of current outputs
OzProj: ompp-win: Digest time 0m 0s
OzProj: ompp-win: Current vs. Reference: 13 the same (of 21)
DIFFERS ===> OzProj: ompp-win: Current vs. Reference: 8 differ
DIFFERS ===> OzProj: ompp-win: Current vs. Reference: DIFFERS: 0_SIGNIFICANT_DIGITS.txt
DIFFERS ===> OzProj: ompp-win: Current vs. Reference: DIFFERS: Experiment1.csv
DIFFERS ===> OzProj: ompp-win: Current vs. Reference: DIFFERS: Experiment2.csv
DIFFERS ===> OzProj: ompp-win: Current vs. Reference: DIFFERS: Experiment3.csv
DIFFERS ===> OzProj: ompp-win: Current vs. Reference: DIFFERS: Experiment4.csv
DIFFERS ===> OzProj: ompp-win: Current vs. Reference: DIFFERS: Experiment5.csv
DIFFERS ===> OzProj: ompp-win: Current vs. Reference: DIFFERS: Experiment6.csv
DIFFERS ===> OzProj: ompp-win: Current vs. Reference: DIFFERS: PersonYearsLived.csv
The specially-named 'result' file 0_SIGNIFICANT_DIGITS.txt
is described elswhere in this topic.
[back to arguments and options]
[back to topic contents]
Unless instructed otherwise, test_models
builds a Release version of a model, without MPI multi-processing capability, and does not save copy of generated C++ code.
This default behaviour can be modified by the following command options:
-
--config
Build configuration: debug or release(default) -
--mpi_processes
Build MPI version (default 0, means no MPI) -
--gencode
Save a copy of the generated C++ code in subfoldergenerated_code
The Debug version of a model is typically considerably slower than the Release version. The Debug version can, however, perform run-time checks which are absent in the Release version. This is particularly true if C++ model code uses assert↗ to verify that expected logical conditions are satisfied at run time (in Debug mode only).
Supply a non-zero value to --mpi_processes
to build an MPI-enabled model (OpenM++ only). The value will be used to launch that number of MPI instances when the model is subsequently run as described in Run. The suffix _mpi
will be appended to the name of the model executable to distinguish it from the normal non-MPI version.
The specialized --gencode option creates a copy of the C++ code generated by the omc compiler (or by the Modgen compiler for the modgen
flavour). If this option is activated, the src
temporary build folder is copied to the output folder generated_code
. The --gencode
option can be used to compare the C++ code generated by different versions of the OpenM++ compiler.
Incidentally, models are capable of multi-threading independent of MPI.
[back to arguments and options]
[back to topic contents]
Unless instructed otherwise, test_models
builds and runs a non-MPI enabled model using the Default scenario. It retains all files from the build, including the model executable and database after the model is run. Also by default, test_models
looks in the model root for a model ini file to use to run the model. The search for the ini file proceeds as follows: If the file test_models.ini
is present, it is used. If test_models.ini
is not present, the file MODEL.ini
is used if present, where MODEL
stands for the name of the model. For more on how test_models
uses a model ini file, see Control - Files.
This default behaviour can be modified by the following command options:
-
--ini
The name of the OpenM++ model ini used when the model is run -
--mpi_processes
The number of mpi processes to launch -
--clean
Remove all build files after the run
For example, if the file OM_ROOT/RiskPaths/Test.ini
exists with content
[OpenM]
SubValues = 16
Threads = 4
[Parameter]
SimulationCases = 16000000
and the following command is issued
perl test_models.pl -m ../models --newref --nomodgen --nocomp --timing --mpi_processes 4 --ini Test.ini RiskPaths
an MPI-enabled version of RiskPaths is built and 4 instances are launched on the workstation under MPI control. The run consists of 16 replicates with 1,000,000 cases in each replicate, for a total of 16,000,000 cases. Each MPI instance will run with 4 threads.
Output is similar to the following:
=========================
test_models 2.0
=========================
Testing: RiskPaths
ompp-win settings: compiler=omc.exe (2021-05-28 02:28 GMT) platform=x64 configuration=release mpi_processes=4
RiskPaths: ompp-win: Deleting previous Reference information
RiskPaths: ompp-win: Build and publish model and Default scenario
RiskPaths: ompp-win: Build time 0m 5s
warning => RiskPaths: ompp-win: 10 build warning(s) - see RiskPaths/test_models/current/ompp-win/logs/build.log
RiskPaths: ompp-win: Run model using Test.ini
RiskPaths: ompp-win: Run time 2m 15s
RiskPaths: ompp-win: Convert outputs (7 digits of precision)
RiskPaths: ompp-win: Convert time 0m 0s
RiskPaths: ompp-win: Create digests of current outputs
RiskPaths: ompp-win: Digest time 0m 0s
RiskPaths: ompp-win: No Reference outputs - create using Current outputs
RiskPaths: ompp-win: Current vs. Reference: 9 the same (of 9)
The truncated log file of the run is similar to the following:
2021-05-31 16:18:57.819 RiskPaths
2021-05-31 16:18:57.819 RiskPaths
2021-05-31 16:18:57.993 Parallel run of 4 modeling processes, 4 thread(s) each
2021-05-31 16:18:57.994 Model build : Windows 64 bit Release
2021-05-31 16:18:57.995 Prepare fixed and missing parameters
2021-05-31 16:18:58.012 Run: 102 Default
2021-05-31 16:18:58.012 Get scenario parameters for process
2021-05-31 16:18:58.013 Model build : Windows 64 bit Release
2021-05-31 16:18:58.013 Prepare fixed and missing parameters
2021-05-31 16:18:58.013 Run: 102 Default
2021-05-31 16:18:58.014 Get scenario parameters for process
2021-05-31 16:18:58.014 member=0 Bind scenario parameters
2021-05-31 16:18:58.014 member=0 Compute derived parameters
2021-05-31 16:18:58.014 member=0 Prepare for simulation
2021-05-31 16:18:58.015 member=1 Bind scenario parameters
2021-05-31 16:18:58.015 member=1 Compute derived parameters
2021-05-31 16:18:58.015 member=1 Prepare for simulation
2021-05-31 16:18:58.015 member=0 Simulation progress=0% cases=0
...
[back to arguments and options]
[back to topic contents]
After running a model, test_models
extracts each output table from the model database and saves a copy of the original data as well as a version rounded to 6 significant digits. test_models
then computes and stores a digest of each rounded csv file which it uses for subsequent comparisons. When a rounded table differs, test_models
verifies that at least one cell differs by more than one part per million using the original table data, before reporting the table as different.
This default behaviour can be modified by the following command options:
-
--significant_digits
The number of significant digits for table result comparison -
--zero_fuzz
Set table cell values to zero if within this value of zero -
--nocells
Disable fallback cell-by-cell verification of differing tables and copy of original data
The number of significant digits used for rounding can be modified using the --significant_digits
option. The default value is 6 (one part in one million). This level of precision can help reduce the number of differing tables reported by test_models while still identifying differences of substantive or logical significance.
Rounding reduces the number of spurious differences which would otherwise be reported by test_models
. It is not unusual for different model versions to produce slightly different results, often at the level of numerical precision (15-16 digits of precision). This can occur due to different C++ compiler optimizations, or by logically equivalent but slightly different algorithms in models. Rounding to less precision reduces the number of spurious differences reported by test_models
.
Most spurious differences are eliminated by rounding, but not all. For example, two numbers can differ only at the 7th digit of precision but round to different values at 6 digits of precision if the two numbers happen to fall on different sides of a 'rounding boundary' (last digit 5). test_models
eliminates these false positives by verifying differing rounded tables using the original unrounded data. In practice, this verification is rapid. This verification can be disabled using the --nocells command line option. The --nocells option also disables the creation of original (unrounded) copies of model output tables.
The value of a table cell can be undefined, eg the mean value of an attribute for a cell with no observations. This is not the same as a table cell value of zero, eg a mean value of 0.0 calculated from one or more observations. test_models
considers an undefined value to be different from the value 0.
Table measures using subtraction can sometimes produce values very close to zero as an artifact of floating-point computations. The presence of such values can produce false positives: tables which test_models
considers different but which are not different substantively. test_models
uses a zero-fuzz
value to handle this situation. If the absolute value of a table cell is within zero_fuzz
of 0.0
, the table cell will be set to 0.0
. The default zero_fuzz
value is 1e-15
. The zero_fuzz
value can be set to 0 to disable zero_fuzz
altogether.
test_models
computes digests using the MD5 cryptographic algorithm. For example, the MD5 digest of the RiskPaths table T04_FertilityRatesByAgeGroup
in the Default
run, rounded to 6 digits and normalized to csv is the hexadecimal value 993ead71da6eed5dde398342278f629a
. The MD5 digest is always 32 hexadecimal digits in length, independent of the size of the input file. Files with different digests are guaranteed to be different, and it is extremely unlikely (read almost impossible) for different files to have the same digest.
[back to arguments and options]
[back to topic contents]
The following options are informational.
-
--help
report usage message and exit -
--version
report test_models version and exit
They have short form versions -h
and -v
.
If either option is on the command line, test_models
will display the requested information and immediately exit.
[back to arguments and options]
[back to topic contents]
The following options affect the normalization of event trace output, if present..
-
--time_format STR
time format for event trace (default 13.6f) -
--modgen_id_offset INT
offset adjustment to Modgen IDs (default 0)
The --time_format
option changes the precision of time in the normalized event trace report. For example, specifying --time_format 19.12f
increases the precision from 6 to 12 digits after the decimal point, producing output like
1969.960131010721 BirthdayEvent (Person 29)
1970.960131010721 timeBirthdayEvent (Person 29)
The --modgen_id_offset
option corrects the entity ID in the Modgen normalized event trace report. This is helpful if the run reproduces a single case from a larger run using a case seed. For such runs, Modgen assigns entity IDs incorrectly, causing spurious differences with ompp event trace output. To fix the issue, the entity IDs in the Modgen event trace report need to be offset by the number of the sub they came from. That sub # may already be known, but if not it can be calculated from the case seed by dividing the case seed by 2^31 and taking the integer part of the result. For example, to determine the sub # of the case seed 11636581014
, divide it by 2147483648
giving 5.4187052948...
. The sub # of the case is 5
. Specifying --modgen_id_offset 5
will offset all entity IDs in the Modgen event trace report by 5, correcting the error and bringing the Modgen event trace report into alignment.
[back to arguments and options]
[back to topic contents]
-
--windows_platform
Windows platform: x64(default) or Win32 -
--modgen_platform
Modgen platform: Win32(default) or x64
If desired, a 32-bit OpenM++ version of a model can be built instead of a 64-bit version, on Windows only, using the --windows_platform
option.
Possibly, a 64-bit version of a Modgen model can be built instead of a 32-bit version using the --modgen_platform
option. The build or run may fail, and run functionality may be limited or compromised.
[back to arguments and options]
[back to topic contents]
=== under construction ===
test_model
operation is controlled mainly by command-line options, but also by the presence and contents of optional specially-named files in the model folder.
OpenM++ model ini files are described
here.
test_models
always enables
Model development options
by automatically specifying the -OpenM.IniAnyKey
to the model when a model ini file is used.
=== under construction ===
test_models
writes a report line by line to the command window where it was invoked as it carries out operations.
test_models
produces two kinds of output: A
report
it writes to the command window where it was invoked, and files which record and persist information about the model runs it performs. The files created by test_folders
can be useful to probe details behind the summary information in a test_models
report.
For example, test_models
reports a count of warnings during model build and also creates the file build.log
containing more information about those warnings.
This subtopic contains the following sections.
- test_models folder: The test_models folder hierarchy and contents
- table outputs: The tables from a model run
- event trace: The normalized event trace from a model run
The output files generated by test_models
are in the subfolder test_models
of the model folder. The test_models
subfolder and hierarchy are created automatically by test_models
as needed. The test_models
subfolder can be deleted at any time, and will be recreated on a subsequent invocation of test_models
.
The following diagram shows the hierarchical structure of the test_models
folder for the RiskPaths
model, with some other subfolders of RiskPaths
shown for context. In this example, test_models
was previously invoked for the three flavours ompp-win
, ompp-linux
, and modgen
.
RiskPaths
├───code
├───modgen
│ ├───bin
│ └───src
├───ompp
│ ├───bin
│ └───src
├───ompp-linux
│ ├───bin
│ └───src
├───parameters
│ └───Default
└───test_models
├───current
│ ├───modgen
│ │ ├───logs
│ │ └───outputs
│ │ └───original
│ ├───ompp-linux
│ │ ├───logs
│ │ └───outputs
│ │ └───original
│ └───ompp-win
│ ├───logs
│ └───outputs
│ └───original
└───reference
├───modgen
│ ├───logs
│ └───outputs
│ └───original
├───ompp-linux
│ ├───logs
│ └───outputs
│ └───original
└───ompp-win
├───logs
└───outputs
└───original
test_models
organizes information using two folder hierarchy levels, as illustrated. For example, RiskPaths
results for the Reference run with the ompp-win
flavour are in the folder RiskPaths/test_models/reference/ompp-win
.
Below is a list of all files produced by test_models
within each flavour folder, eg within RiskPaths/current/ompp-win
:
-
tombstone.txt
- Contains basic information about the compiler, members, ini file, etc. used to build and run the model.test_models
reports if this file differs between Current and Reference. -
logs/build.log
- A copy of captured console output from all stages of the build process. -
logs/run.log
- A copy of the log file produced by the model when run. -
outputs/trace.txt
- A normalized version of the event trace file, if produced by the model. -
outputs/0_MODEL_INI.txt
- The contents of the model ini file used to run the model, if any. -
outputs/0_SIGNIFICANT_DIGITS.txt
- Contains the number of significant digits used to round model output tables before computing digests. -
outputs/digests.txt
- Contains the name and digest value of each file in the outputs folder (excluding subfolders). -
outputs/*.csv
- Model output tables, rounded. -
outputs/original/*.csv
- Model output tables, original (unrounded).
[back to output files]
[back to topic contents]
Each table is saved in a flat csv format with a header line. Zero-based indexes identify the table cell. Each row consists of all classificatory dimension indices, followed by the single measure dimension index, followed by the cell value. The order of classificatory dimensions is the same as the order in the table
statement which declared the table in model code. Classificatory dimensions will include an additional trailing level for the margin, if a margin is specified for the dimension in the table declaration. The measure index is always present even if there is only one measure in the measure dimension. The cell value can be empty. Here's an example extract:
Dim0,Dim1,Dim2,Dim3,Value
0,0,0,0,258.453
0,0,0,1,1119.96
0,0,0,2,933.714
0,0,0,3,86.1511
0,0,1,0,172.302
...
[back to output files]
[back to topic contents]
=== under construction ===
Note that times >= 32767 will be shown as 99999, as will +inf, to improve comparability between ompp and Modgen event trace files.
Note options which control normalized event trace operation.
Ompp models must set EventTrace.Style to modgen for event trace files to be normalized and comparable.
[back to output files]
[back to topic contents]
This example illustrates how to arrange that the default test_models
run is fast, and is easier to debug or investigate if run-time anomalies or unexpected outputs occur during model development. The underlying idea is to develop a model in small incremental steps, and use test_models
to "fail fast, fail early", after each incremental change to the model. This approach is efficient only if the testing phase is fast. This example uses the OncoSimX
model on the Windows platform.
Step 1 investigates how the model performs with a normal Default run. First, a fresh version of the model is created using git to clone from the git server to the temporary folder C:/Temp/OncoSimX
. Next, test_models
is run with the --timing
option to evaluate the performance of the Default run.
C:\temp\OncoSimX>%OM_ROOT%\bin\test_models -m .. --nomodgen --nocomp --timing OncoSimX
=========================
test_models 2.1
=========================
Testing: OncoSimX
ompp-win settings: compiler=omc.exe (2021-06-29 06:31 GMT) platform=x64 configuration=release
OncoSimX: ompp-win: Build and publish model and Default scenario
OncoSimX: ompp-win: Build time 1m 47s
warning => OncoSimX: ompp-win: 1794 build warnings - see OncoSimX/test_models/current/ompp-win/logs/build.log
OncoSimX: ompp-win: Run model using OncoSimX.ini
OncoSimX: ompp-win: Run time 25m 3s
OncoSimX: ompp-win: Convert outputs (6 digits of precision)
OncoSimX: ompp-win: Convert time 1m 33s
OncoSimX: ompp-win: Create digests of current outputs
OncoSimX: ompp-win: Digest time 0m 0s
OncoSimX: ompp-win: No Reference outputs - create using Current outputs
OncoSimX: ompp-win: Current vs. Reference: 215 the same (of 215)
OncoSimX: ompp-win: Compare time 0m 0s
The test_models
report shows that the run itself took 25m, and that the run used the model ini file OncoSimX.ini
. An ini file with the same name as the model is used by default if it exists. Here's the contents of that file:
[OpenM]
SubValues = 12
Threads = 6
[Parameter]
SimulationCases = 1000000
SimulationSeed = 16807
; Complete example of ini-file located at: props/model/ompp/Model-example.ini
The run had 1,000,000 cases divided among 12 members (aka replicates or sub-samples), and was accelerated by running 6 threads in parallel (the workstation for the run had 8 processor cores). As a test, 1,000,000 cases does explore many conditions in the model and can produce statistically meaningful results, but a run of that size with OncoSimX
is not suitable for quick iteration during model development. Moreover, a multi-threaded run is much more difficult to debug than a single-threaded run, making it a poor choice for testing during development. Ignoring run time, test_models
took about 3m 15s. If the run were modified to take 1m 45s of run time instead, with one thread of execution, the total time would be about 5m, which would be acceptable for rapid testing.
Step 2 creates a new model ini file named test_models.ini
based on the original model ini file OncoSimX.ini
with several modifications. The number of members is changed to 1, the number of threads to 1, and the the number of cases changed somewhat arbitrarily to 5,000 to see how fast that will turn out:
[OpenM]
SubValues = 1
Threads = 1
[Parameter]
SimulationSeed = 1
SimulationCases = 5000
The choice of file name for the modified model ini file was not arbitrary. test_models
will use by default a model ini file named test_models.ini
if it exists.
Step 3 runs test_models
with the new model ini test_models.ini
which it will use by default. The --newref
option is used to replace the previous results to avoid reporting meaningless differences.
C:\temp\OncoSimX>%OM_ROOT%\bin\test_models -m .. --nomodgen --nocomp --timing --newref OncoSimX
=========================
test_models 2.1
=========================
Testing: OncoSimX
ompp-win settings: compiler=omc.exe (2021-06-29 06:31 GMT) platform=x64 configuration=release
OncoSimX: ompp-win: Deleting previous Reference information
OncoSimX: ompp-win: Build and publish model and Default scenario
OncoSimX: ompp-win: Build time 1m 46s
warning => OncoSimX: ompp-win: 1794 build warnings - see OncoSimX/test_models/current/ompp-win/logs/build.log
OncoSimX: ompp-win: Run model using test_models.ini
OncoSimX: ompp-win: Run time 1m 41s
OncoSimX: ompp-win: Convert outputs (6 digits of precision)
OncoSimX: ompp-win: Convert time 1m 19s
OncoSimX: ompp-win: Create digests of current outputs
OncoSimX: ompp-win: Digest time 0m 1s
OncoSimX: ompp-win: No Reference outputs - create using Current outputs
OncoSimX: ompp-win: Current vs. Reference: 215 the same (of 215)
OncoSimX: ompp-win: Compare time 0m 0s
The report shows that test_models.ini
was used as intended. The total time for test_models
to build, run, and compare results was 4m 47s, which is about what was targeted as a quick test for incremental model development.
Step 4 uses git to add test_models.ini
to the local repository in C:/Temp/OncoSimX
, and then push the change to the server repository to persist it and make it available to others on the modelling team.
Step 5 runs test_models
a final time using test_models.ini
to see how long the Modgen version takes.
C:\Development\X\ompp\Perl>perl test_models.pl -m ../../models --noompp --timing OncoSimX
=========================
test_models 2.1
=========================
Testing: OncoSimX
modgen settings: version=12,1,3,0 (2019-12-19 20:31 GMT) platform=Win32 configuration=release
OncoSimX: modgen: Build model and prepare Default scenario
OncoSimX: modgen: Build time 0m 15s
warning => OncoSimX: modgen: 1 build warnings - see OncoSimX/test_models/current/modgen/logs/build.log
OncoSimX: modgen: Run model using test_models.ini
OncoSimX: modgen: Run time 38m 54s
warning => OncoSimX: modgen: 1 run warnings - see OncoSimX/test_models/current/modgen/logs/run.log
OncoSimX: modgen: Convert outputs (6 digits of precision)
OncoSimX: modgen: Convert time 12m 3s
OncoSimX: modgen: Create digests of current outputs
OncoSimX: modgen: Digest time 0m 1s
OncoSimX: modgen: Current vs. Reference: 205 the same (of 205)
OncoSimX: modgen: Compare time 0m 0s
Even with the small run in test_models.ini
, the Modgen version takes 50m 13s, making it not suitable for quick development tests for the OncoSimmX
model with all tables. The Modgen version simulates quickly, but writing all table outputs and converting those outputs takes a long time. The reason is that Modgen uses a Microsoft Access database for storing tables, and MS Access is slow.
The folder C:/Temp/OncoSimX
is now deleted, having served its purpose.
In this example, a new OpenM++ release has been announced, and a model developer is asked to verify that the team's model successfully builds and runs with the new version and produces identical results. The model developer uses a Windows workstation.
Step 1 downloads and extracts the new version of OpenM++, released on 2021-06-29, to a new folder on the workstation. To simplify changing between OpenM++ versions, the model developer previously created a folder named C:/ompp_versions
on the workstation. A new command prompt is opened and used to verify the contents of that folder after the download and extraction:
C:\ompp_versions>dir
Volume in drive C is OS
Volume Serial Number is 14E2-D15F
Directory of C:\ompp_versions
2021-06-29 07:27 AM <DIR> .
2021-06-29 07:27 AM <DIR> ..
2021-06-12 09:49 PM <DIR> openmpp_win_20210505
2021-06-12 09:50 PM <DIR> openmpp_win_20210602
2021-06-12 09:50 PM <DIR> openmpp_win_20210612
2021-06-29 07:27 AM <DIR> openmpp_win_20210629
0 File(s) 0 bytes
6 Dir(s) 1,717,655,265,280 bytes free
A command is issued to verify the version of OpenM++ currently in use on the workstation:
C:\ompp_versions>echo %OM_ROOT%
C:\ompp_versions\openmpp_win_20210602
The version currently in use is 2021-06-02 which is two versions old because the development team decided to skip the mid-month OpenM++ bug-fix release because that bug was known not to affect the team's model.
Step 2 prepares the test. The OncoSimX
model is used in this example and it's assumed that the model developer is working actively on the model and has unchecked code changes. To not interfere with that current work-in-progress, git is used to create a new temporary clone of the stable trunk version of OncoSimX
in the temporary folder C:/Temp/OncoSimX
.
Step 3 invokes test_models
to create a Reference run using the team's working version (2021-06-02) of OpenM++:
C:\ompp_versions>cd C:\Temp\OncoSimX
C:\temp\OncoSimX>perl %OM_ROOT%/perl/test_models.pl -m .. --nomodgen --nocomp OncoSimX
=========================
test_models 2.0
=========================
Testing: OncoSimX
ompp-win settings: compiler=omc.exe (2021-06-02 03:23 GMT) platform=x64 configuration=release
OncoSimX: ompp-win: Build and publish model and Default scenario
warning => OncoSimX: ompp-win: 1794 build warning(s) - see OncoSimX/test_models/current/ompp-win/logs/build.log
OncoSimX: ompp-win: Run model using test_models.ini
OncoSimX: ompp-win: Convert outputs (7 digits of precision)
OncoSimX: ompp-win: Create digests of current outputs
OncoSimX: ompp-win: No Reference outputs - create using Current outputs
OncoSimX: ompp-win: Current vs. Reference: 215 the same (of 215)
The -m ..
option specifies that the folder containing models is the parent folder of the current folder.
Step 4 switches to the new OpenM++ release dated 2021-06-29 by changing the OM_ROOT
environment variable:
C:\temp\OncoSimX>set OM_ROOT=C:\ompp_versions\openmpp_win_20210629
and invokes test_models
again using the same arguments:
C:\temp\OncoSimX>perl %OM_ROOT%/perl/test_models.pl -m .. --nomodgen --nocomp OncoSimX
=========================
test_models 2.1
=========================
Testing: OncoSimX
ompp-win settings: compiler=omc.exe (2021-06-29 06:31 GMT) platform=x64 configuration=release
OncoSimX: ompp-win: Build and publish model and Default scenario
warning => OncoSimX: ompp-win: 1794 build warnings - see OncoSimX/test_models/current/ompp-win/logs/build.log
OncoSimX: ompp-win: Run model using test_models.ini
OncoSimX: ompp-win: Convert outputs (6 digits of precision)
OncoSimX: ompp-win: Create digests of current outputs
OncoSimX: ompp-win: Current vs. Reference: tombstone info differs:
OncoSimX: ompp-win: Reference: compiler=omc.exe (2021-06-02 03:23 GMT) platform=x64 configuration=release members=1
OncoSimX: ompp-win: Current: compiler=omc.exe (2021-06-29 06:31 GMT) platform=x64 configuration=release members=1
OncoSimX: ompp-win: Current vs. Reference: 214 the same (of 215)
DIFFERS ===> OncoSimX: ompp-win: Current vs. Reference: 1 differ
DIFFERS ===> OncoSimX: ompp-win: Current vs. Reference: DIFFERS: 0_SIGNIFICANT_DIGITS.txt
test_models
reports that the OpenM++ compiler versions for the Reference and Current runs differ, as expected. Unexpectedly, it reported a single differing file. The name of that file and the two test_models
reports pinpoint the cause: The default number of digits used by test_models
was changed from 7 digits to 6 digits between the two OpenM++ releases. The change in the number of significant digits turned out not to affect the rounded versions of the model output tables in the two runs.
The test, at this point, has established that the model builds and runs successfully with the new version of OpenM++ and produces identical outputs for the small fast run used normally by test_models
during OncoSimX
development.
In step 5, the model developer decides to perform a more demanding verification by comparing results between the two OpenM++ releases using a larger run of 1,000,000 cases and 12 replicates. These are the settings in the model ini file OncoSimX/OncoSimX.ini
:
[OpenM]
SubValues = 12
Threads = 6
[Parameter]
SimulationCases = 1000000
SimulationSeed = 16807
; Complete example of ini-file located at: props/model/ompp/Model-example.ini
The OpenM++ version is switched back to the older 2021-06-02 version
C:\temp\OncoSimX>set OM_ROOT=C:\ompp_versions\openmpp_win_20210602
and test_models
is invoked, this time explicitly specifying the number of significant digits as well as the model ini file to use. The --newref
option is used to create a new Reference run using the larger run specified in the ini file.
C:\temp\OncoSimX>perl %OM_ROOT%/perl/test_models.pl -m .. --nomodgen --nocomp --significant_digits 6 --ini OncoSimX.ini --newref OncoSimX
=========================
test_models 2.0
=========================
Testing: OncoSimX
ompp-win settings: compiler=omc.exe (2021-06-02 03:23 GMT) platform=x64 configuration=release
OncoSimX: ompp-win: Deleting previous Reference information
OncoSimX: ompp-win: Build and publish model and Default scenario
warning => OncoSimX: ompp-win: 1794 build warning(s) - see OncoSimX/test_models/current/ompp-win/logs/build.log
OncoSimX: ompp-win: Run model using OncoSimX.ini
OncoSimX: ompp-win: Convert outputs (6 digits of precision)
OncoSimX: ompp-win: Create digests of current outputs
OncoSimX: ompp-win: No Reference outputs - create using Current outputs
OncoSimX: ompp-win: Current vs. Reference: 215 the same (of 215)
The OpenM++ version is switched once again to the new release 2021-06-29
C:\temp\OncoSimX>set OM_ROOT=C:\ompp_versions\openmpp_win_20210629
and test_models
is run again to produce a comparable Current run and compare it to the Reference run:
C:\temp\OncoSimX>perl %OM_ROOT%/perl/test_models.pl -m .. --nomodgen --nocomp --significant_digits 6 --ini OncoSimX.ini OncoSimX
=========================
test_models 2.1
=========================
Testing: OncoSimX
ompp-win settings: compiler=omc.exe (2021-06-29 06:31 GMT) platform=x64 configuration=release
OncoSimX: ompp-win: Build and publish model and Default scenario
warning => OncoSimX: ompp-win: 1794 build warnings - see OncoSimX/test_models/current/ompp-win/logs/build.log
OncoSimX: ompp-win: Run model using OncoSimX.ini
OncoSimX: ompp-win: Convert outputs (6 digits of precision)
OncoSimX: ompp-win: Create digests of current outputs
OncoSimX: ompp-win: Current vs. Reference: tombstone info differs:
OncoSimX: ompp-win: Reference: compiler=omc.exe (2021-06-02 03:23 GMT) platform=x64 configuration=release members=12
OncoSimX: ompp-win: Current: compiler=omc.exe (2021-06-29 06:31 GMT) platform=x64 configuration=release members=12
OncoSimX: ompp-win: Current vs. Reference: 215 the same (of 215)
The larger run executed successfully with the new OpenM++ release, and all output results were the same to 6 significant digits.
In this example, the OM_ROOT
environment variable was changed using the set
command (not the setx
command) to select an OpenM++ version. The changed value of OM_ROOT
applied only to commands issued in the command prompt window used for the test. The value of OM_ROOT
in other command prompt windows or Visual Studio sessions was unaffected by the test.
=== under construction ===
=== under construction ===
In this example, test_models
is used to build and run Debug versions of a model, using both OpenM++ and Modgen versions. Debug versions run more slowly but contain more run-time checks which can identify errors, so it is a good idea to test the Debug versions from time to time. test_models1 makes that easy. This example uses the
OncoSimX` model running on a Windows system.
In step 1, a fresh set of Reference runs are created:
The Modgen run took considerably longer than the OpenM++ run.
In this example, test_models
is used to evaluate the performance difference between two versions of the same model, holding other effects constant. Specifically, this example seeks to measure the performance cost of the run-time checks which OpenM++ performs to ensure that attributes are not modified directly or indirectly in event time functions. Although those runtime checks are core OpenM++ functionality, the OpenM++ option verify_attribute_modification
can be used to remove them.
Step 1 sets up the experiment. The OncoSimX
model is chosen because it is a large-scale production model with many events. The experiment is done on a secondary (older) Windows workstation which will run no other tasks during the experiment. The experiment is temporary, so a single git command is issued to create a new clone of OncoSimX
in a convenient temporary location, which in this example is C:/Temp/OncoSimX
. The OncoSimX
clone contains no test_models
results, which is appropriate for a model git repository. It does contain the model ini file test_models.ini
in the model root.
Step 2 sets up the test_models
model run to serve as a performance test. Normally, a model run for test_models
is small and fast. This experiment requires a run which is large enough to produce reliable performance information. The OncoSimX
model ini file test_models.ini
produces small fast runs for rapid development. The contents are:
[OpenM]
SubValues = 1
Threads = 1
[Parameter]
SimulationSeed = 1
SimulationCases = 5000
[Tables]
;Retain = \
; TG01_Breast_Cancer_Tables
A single line is modified to increase the number of cases from 5,000 to 1,000,000 for the experiment.
SimulationCases = 1000000
Step 3 uses test_models
to create a Reference run to serve as a basis for comparison. A command window is opened and test_models
is invoked:
perl test_models.pl -m C:/Temp --timing --nomodgen OncoSimX
The temporary OncoSimX
clone was created in the folder C:/Temp
, so that folder is specified as the models folder using the -m
option. Timing information is requested using the --timing
option. The experiment is being done on a Windows workstation with Modgen installed, so the --nomodgen
argument is used to suppress that flavour.
The output is
=========================
test_models 2.1
=========================
Testing: OncoSimX
ompp-win settings: compiler=omc.exe (2021-07-01 12:27 GMT) platform=x64 configuration=release
OncoSimX: ompp-win: Build and publish model and Default scenario
OncoSimX: ompp-win: Build time 3m 58s
warning => OncoSimX: ompp-win: 1794 build warnings - see OncoSimX/test_models/current/ompp-win/logs/build.log
OncoSimX: ompp-win: Run model using test_models.ini
OncoSimX: ompp-win: Run time 98m 8s
OncoSimX: ompp-win: Convert outputs (6 digits of precision)
OncoSimX: ompp-win: Convert time 2m 10s
OncoSimX: ompp-win: Create digests of current outputs
OncoSimX: ompp-win: Digest time 0m 1s
OncoSimX: ompp-win: No Reference outputs - create using Current outputs
OncoSimX: ompp-win: Current vs. Reference: 215 the same (of 215)
OncoSimX: ompp-win: Compare time 0m 0s
OncoSimX: Flavour compare time 0m 0s
Note that test_models
created Reference results from Current results because no Reference run was present. The reported run time was 98m 8s.
Step 4 modifies the model to remove the normal run-time checks for the experiment. Because this is a temporary clone of OncoSimX which will be discarded after the experiment, the source code can be modified freely with no consequences. The small source file TraceOptions.mpp
was selected arbitrarily to implement the experiment, and the following line added:
options verify_attribute_modification = off;
Step 5 uses test_models
to create a Current run which will be compared to the Reference run. The invocation of test_models
is identical to the previous invocation.
perl test_models.pl -m C:/Temp --timing --nomodgen OncoSimX
The output is
=========================
test_models 2.1
=========================
Testing: OncoSimX
ompp-win settings: compiler=omc.exe (2021-07-01 12:27 GMT) platform=x64 configuration=release
OncoSimX: ompp-win: Build and publish model and Default scenario
OncoSimX: ompp-win: Build time 3m 58s
warning => OncoSimX: ompp-win: 1794 build warnings - see OncoSimX/test_models/current/ompp-win/logs/build.log
OncoSimX: ompp-win: Run model using test_models.ini
OncoSimX: ompp-win: Run time 95m 16s
warning => OncoSimX: ompp-win: 1 run warnings - see OncoSimX/test_models/current/ompp-win/logs/run.log
OncoSimX: ompp-win: Convert outputs (6 digits of precision)
OncoSimX: ompp-win: Convert time 2m 12s
OncoSimX: ompp-win: Create digests of current outputs
OncoSimX: ompp-win: Digest time 0m 0s
OncoSimX: ompp-win: Current vs. Reference: 215 the same (of 215)
OncoSimX: ompp-win: Compare time 0m 0s
OncoSimX: Flavour compare time 0m 0s
All model outputs are identical as expected. test_models
reports one run-time warning which was absent in the first run. That's because OpenM++, by design, emits a warning on every run of a model which disables run-time checking of event times. test_models
noted the warning and reported it when it analyzed the run log. The reported run time was 95m 16s.
Step 6 compares performance between the Reference and Current runs. The model run time in the Current run was 172s less than the Reference run, and shows that about 2.9% of the elapsed time of the Reference run was overhead associated with the run-time checks. That is a realistic assessment of the overhead of the run-time checks for OncoSimX
when used in production.
The elapsed time reported by test_models
includes the time required to read parameters and to assemble and write tables, which can be significant. It might also be interesting to assess the performance effect on the simulation time alone, removing the fixed cost of parameter reading and table assembly and writing. The model log files saved by test_models
can be used to perform that calculation.
Here's a line extracted from the Reference log file, located at
C:/Temp/OncoSimX/test_models/reference/ompp-win/logs/run.log
:
2021-07-01 10:16:52.928 member=0 Simulation summary: cases=1000000, events/case=1128.9, entities/case=5.4, elapsed=5693.747633s
and here's the corresponding line extracted from the Current log file, located at
C:/Temp/OncoSimX/test_models/current/ompp-win/logs/run.log
.
2021-07-01 12:31:45.777 member=0 Simulation summary: cases=1000000, events/case=1128.9, entities/case=5.4, elapsed=5525.430919s
Conveniently, this line in the OpenM++ model log reports the elapsed time of the computational phase, in seconds. The simulation time was 5694s for the Reference run and 5525s for the Current run, for a difference of 168s. So about 3.0% of the computation time was overhead associated with the run-time checks. That's only slightly higher than the proportion computed using elapsed time because the time required to read parameters and to assemble and write tables (about 190s) is a relatively small portion of the elapsed time of an OncoSimX
run of 1,000,000 (5525s).
=== under construction ===
The following applies to a case-based model.
If test_models detects unexpected output differences which are not immediately explained by recent model code changes or by perusing differences in specific tables, it can be useful to determine whether the differences are due to a handful of different cases in the run, or to some other cause.
To find out, turn on the case checksum option by adding the statement
options case_checksum = on;
to model code. In the RiskPaths example model, this is done in a dedicated model source file named TraceOptions.mpp
whcih is normally:
//LABEL(TraceOptions, EN) Control options for trace output
/* NOTE(TraceOptions, EN)
The options must be set to off for normal use.
If the options are on, very large trace output files may be produced by the model.
*/
options event_trace = off;
options case_checksum = off;
Turning case_checksum
on will cause a trace file to be generated when test_models runs the model which contains one line for each case. For RiskPaths, the first lines of the trace file look like this:
Seed: 1 Sample: 0 Checksum: 4.4094661700e+02
Seed: 470583131 Sample: 0 Checksum: 4.1293225500e+02
Seed: 1715377866 Sample: 0 Checksum: 3.4860055300e+02
Seed: 430166727 Sample: 0 Checksum: 1.3482143400e+02
Seed: 743781969 Sample: 0 Checksum: 1.4831547800e+02
Seed: 594673803 Sample: 0 Checksum: 4.1803896100e+02
Seed: 2048386738 Sample: 0 Checksum: 3.5386231900e+02
Seed: 1681919254 Sample: 0 Checksum: 4.0119902000e+02
Seed: 78320563 Sample: 0 Checksum: 3.9439706500e+02
Seed: 1637697257 Sample: 0 Checksum: 4.9065368400e+02
Seed: 127732046 Sample: 0 Checksum: 9.8715092900e+02
Seed: 2114816392 Sample: 0 Checksum: 3.9576701300e+02
Seed: 1616242686 Sample: 0 Checksum: 1.2478963010e+03
Seed: 1222943222 Sample: 0 Checksum: 4.5861339400e+02
Seed: 170881636 Sample: 0 Checksum: 1.4320257000e+02
This file is noticed by test_models and will participate in test_models comparisons. Test_models saves the trace file under the name trace.txt
in the same folder where it places csv versions of tables.
If test_models
reports that the file trace.txt
differs between the ompp and modgen versions, then one or more cases differ in the two flavours. Use a tool or text editor to compare the two versions of trace.txt
to determine the seed of the first case which has a differing checksum in the two variants.
More to follow...