OpenMpp Development Notes - openmpp/openmpp.github.io GitHub Wiki

This page contains various notes only for OpenM++ developers. There is no useful information on that page for anyone else. It is a notes, they are not in any specific order and may not true. OK, you have been warned.

OpenM++ consists of 6 source code repositories published at GitHub / openmpp. Core portion of openM++ located at GitHub / openmpp / main and has following structure:

• bin - used for OpenM++ compiled binaries and third party tools
• include - includes for public interfaces of compiler and libraries
  • libopenm - model runtime library public interface
  • omc - model compiler public interface
• licenses - third party lincences
• models - test models, for example:
  • NewCaseBased - simple test model
  • NewTimeBased - simple test model
  • modelOne - test model for runtime library, does not use OpenM++ compiler
• openm - OpenM++ core source code
  • libopenm - model runtime library (libopenm) and compiler library (libopenm_omc_db)
    • common - common helper routines, for example: log
    • db - data access classes
    • include - includes for libopenm and libopenm_omc_db
    • model - model base classes
    • msg - message passing library
  • main.cpp - models main() entry point
  • libsqlite - SQLite with extension functions such as SQRT()
  • omc - OpenM++ compiler
• Perl - perl scripts
• props - VC++ project includes to build the models
• R - openMpp R library: integration between OpenM++ and R
• sql - sql scripts to create openM++ database
  • db2 - DB2 version of openM++ database scripts
  • mssql - Microsoft SQL Server version of openM++ database scripts
  • mysql - MySql version of openM++ database scripts
  • postgresql - PostgreSql version of openM++ database scripts
  • sqlite - SQLite version of openM++ database scripts

As it is now model executable output log messages into three streams:

• standard output (console)
• "last" log file: /current/working/dir/modelExeName.log
• "stamped" log file: /current/working/dir/modelExeName.date_time.pid.log

Model trace output is similar to log output but works much faster. Trace output is buffered and may be lost if something goes wrong and model crushed.

You can adjust output log and trace output inside of main() by changing: theLog->init(...); parameters. It is also be controlled by .ini options file.

You may need to change defines to build OpenM++ from source code:

• OM_DB_SQLITE: use SQLite as database provider (only one supported at the moment)
• OM_MSG_MPI: use MPI as for message passing library (see below)
• OM_MSG_EMPTY: use empty version message passing library (default value)
• OM_UCVT_MSSTL: use c++11 STL to convert strings to UTF-8 (default on Windows)
• OM_UCVT_ICONV: use glibc iconv to convert strings and file content to UTF-8 (default on Linux)

Please note:

• OM_MSG_MPI and OM_MSG_EMPTY mutually exclusive
• to set defines properly change openm.build.props (on Windows) or use make OM_MSG_USE=MPI (on Linux)
• OM_UCVT_MSSTL and OM_UCVT_ICONV mutually exclusive
• OM_UCVT_MSSTL tested on Windows with VC++2012 and account for Microsoft-specific implementation of STL codecvt classes.

Defines to compile libsqlite library with extension functions: SQLITE_ENABLE_COLUMN_METADATA; SQLITE_OMIT_LOAD_EXTENSION; HAVE_ACOSH; HAVE_ASINH; HAVE_ATANH;

To avoid innumerous compatibility errors and warnings following must be defined: _CRT_SECURE_NO_WARNINGS and _CRT_NONSTDC_NO_WARNINGS.

IDbExec interface is db-connection wrapper and only the place where real SQL operations executed. All other classes are to wrap OpenM++ database tables and implement "business logic".

Data library is NOT thread-safe by design, do not pass it objects between model threads without proper guards.

Difference between OpenM++ database schema and Modgen schema:

• support multiple models and multiple versions of the same model
• support multiple run results of each model
• tends to be more "relational", i.e.:
  • language-specific rows moved to separate tables
  • sub-samples are in rows not in columns

Database schema "read-only" compatible with Modgen database. For each Modgen table corresponding view created which allow to read from OpenM++ database as from Modgen database. If OpenM++ database contains multiple models (or versions) then it not be exposed to Modgen compatibility views.

If database connection string is not specified then model try to open SQLite database with name ModelName.sqlite (i.e.: modelOne.sqlite) in current working directory. Other word, default database connection strig is:

Database=ModelName.sqlite; Timeout=86400; OpenMode=ReadWrite;

Database can be created by following commands:

cd
sqlite3 ModelName.sqlite < ../sql/sqlite/create_db_sqlite.sql
sqlite3 ModelName.sqlite < ModelName_create_model.sql
sqlite3 ModelName.sqlite < ModelName_insert_parameters.sql

On Linux slite3 executable most likely in your PATH. On Windows you must download sqlite3.exe from SQLite web-site.

Following parameters allowed for SQLite database connection:

• Database - (required) database file name or URI, file name can be empty
• Timeout - (optional) table lock "busy" timeout in seconds, default=0
• OpenMode - (optional) database file open mode: ReadOnly, ReadWrite, Create, default=ReadOnly
• DeleteExisting - (optional) if true then delete existing database file, default: false

If OpenMode=Create specified then database file created if not exist, which is default SQLite behavior.

Note: minimal connection string syntax for SQLite provider is: "Database=" and in that case SQLite will open temporary database. That kind of connection string does not really make sense for OpenM++ models because temporary database will be deleted after model exit.

Message passing library (a.k.a. execute library) used for:

• broadcast metadata and input parameters from root process to slave modeling processes
• gather output modeling results from all modeling processes into root process

That library has two versions:

• define OM_MSG_MPI: MPI-based version which does the job as described above (MPI component must be installed)
• define OM_MSG_EMPTY: empty version of library, which does nothing and don't required anything installed

When empty version of library can useful?

To develop and debug your model without having MPI installed and without complexity of multi-process debugging. Obviously, some technique must be used to debug modeling logic inside of single process.

IMsgExec interface is main class for message passing library. All processes involved in the modeling must can be identified by integer process rank. Root process rank is zero.

Messaging library is NOT thread-safe, at least for now, do not pass it objects between model threads without proper guards. It may change in the future versions.

All strings inside of openM++ source code expected to be are UTF-8 encoded. If you need to pass string to openM++ API, please convert it to UTF-8 first. There is helper function which return file content converted as UTF-8 string:

string fileContent = fileToUtf8("someFile.txt");

Following rules applied to detect file encoding:

• if byte order mark (BOM) present in the file then it converted according to BOM
• if first 2048000 bytes of file are UTF-8 then file considered as UTF-8 and not converted
• if code page (encoding name) specified, i.e.: "English_US.1252" then it used for conversion
• default user code page (encoding name) used to convert file content to UTF-8

You can use optional parameter to explicitly specify code page (encoding name):

string fileContent = fileToUtf8("someFile.txt", "English_Canada.1252"); // Windows: CP-1252
string fileContent = fileToUtf8("someFile.txt", "WINDOWS-1252");        // Linux:   CP-1252

Encoding name is OS-specific and conversion would fail if name is invalid.

Note: conversion from UTF-32 to UTF-8 not supported on Windows.

OpenM++ is using MD5 digest to compare and find models, parameters, output tables and types in database. There are two digests calculated for model run:

• model run values digest which based on
  • values in model run output tables
  • values of model run input parameters
• model run metadata digest which is unique key of model run Model run values digest calculated only after run is completed. It can be empty if run failed.

Model run results do include output table values and all input parameter values. Model runs are stored in database as single copy only. For example, if digest of (parameter A value of model run 101) == digest of (parameter A value of model run 123) then only value from run 101 actually stored in database and run 123 is a link to run 101 value.

Following rules are used to calculate digests:

Model digest:
-------------
model name, model type, model version
for all model types:
  type digest
for all model parameters:
  parameter digest
for all model output tables:
  table digest

Parameter digest:
----------------
parameter name, rank, type digest
for all dimensions:
  id, name, size, type digest

Output table digest:
--------------------
table name, rank
for all dimensions:
  id, name, size (including "total" item), type digest
for all accumulators:
  acc id, name, source
  examples:
    id:     1
      name:   acc1
      source: accumulator 1: sum(delta(interval(duration(smoking_status, NON_SMOKER))))
    id:     9
      name:   Expr4
      source: 1.0E2 * ( acc4 / acc0 )
for all expressions (a.k.a. measures):
  id, name, source
  examples:
    id:     0
      name:   Expr0
      source: ( OM_AVG( acc0 ) / ( OM_AVG( acc1 ) - OM_AVG( acc2 ) ) )
    id:     8
      name:   E8
      source: OM_AVG(acc8)

Type digest:
------------
type name, dictionary id (e.g.: 3=range), "total" enum id
for all enums:
   id, enum name

Import digest for parameter or output table:
--------------------------------------------
rank, type digest
for all dimensions:
  id, name, size, type digest

Model run metadata digest:
--------------------------
model digest, run name, sub-values count, create date-time, run stamp

Model run value digest:
-----------------------
sub-values count, completed sub-values count, run status

for all parameters:
  parameter value digest

for all output tables:
  output table value digest

Value digest for parameters:
----------------------------
parameter_name, parameter_digest
sub_id, dimension names, param_value as comma separated header
  example (2 dimensions):
    sub_id,dim0,dim1,param_value
for all value rows:
  select sub_id, dimensions id, param_value
  convert sub_id, dimensions id into strings
  convert param_value to string
    if type is float then format as %.15g
    if type is boolean then "true" or "false"
  example (2 dimensions boolean):
    2,11,22,true

Value digest for output table:
------------------------------
table_name, table_digest

for all accumulators:
  accumulators value digest

for all expressions:
  expressions value digest

Value digest for output table accumulators:
-------------------------------------------
comma separated header: acc_id, sub_id, dimension names, acc_value
  example (2 dimensions):
    acc_id,sub_id,dim0,dim1,acc_value

for all value rows:
  select acc_id, sub_id, dimensions id, acc_value
  convert acc_id, sub_id, dimensions id into strings
  format acc_value as %.15g
  example (2 dimensions):
    2,15,11,22,0.1234

Value digest for output table expressions:
------------------------------------------
comma separated header: expr_id, dimension names, expr_value
  example (4 dimensions):
    expr_id,dim0,dim1,dim2,dim3,expr_value

for all value rows:
  select expr_id, sub_id, dimensions id, expr_value
  convert expr_id, sub_id, dimensions id into strings
  format expr_value as %.15g
  example (4 dimensions):
    1,11,22,33,44,0.789