Oms in cloud - openmpp/openmpp.github.io GitHub Wiki

OpenM++ web-service (oms) can provide basic computational resources management for your local computer or cluster of servers on local network or in cloud. It can manage model runs queue if your computational resources (CPU and memory) are limited and also can automatically start and stop cloud servers.

Content below assuming you are familiar with basics of Oms: openM++ web-service.

For cloud and clusters configuration examples please see Oms: Cloud and cluster setup.

If you want to have model runs queue, or using openM++ in cloud and want automatically scale up and down cloud resources, (e.g. start and stop virtual machines for model runs) then start oms with job control option:

oms -oms.JobDir job

Following directory structure expected if user is running OpenM++ on local computer:

./        -> oms "root" directory, by default it is current directory
    html/    -> web-UI directory with HTML, js, css, images...
             disk.ini  -> (optional) disk storage usage settings and shared models library
    etc/     -> config files directory, contain template(s) to run models
    log/     -> recommended log files directory
    models/
          bin/  -> default model.exe and model.sqlite directory
          log/  -> default directory for models run log files
          doc/  -> models documentation directory
          home/ -> user personal home directory
              io/download  -> user directory for download files
              io/upload    -> user directory to upload files
    job/  -> model run jobs control directory
          job.ini   -> job control settings
          active/   -> active model run state files
          history/  -> model run history files
          past/     -> (optional) shadow copy of history folder, invisible to the end user
          queue/    -> model run queue files
          state/    -> jobs state and computational servers state files
               jobs.queue.paused      -> if such file exists then jobs queue is paused
               jobs.queue.all.paused  -> if such file exists then all jobs in all queues are paused

Model runs queue and computational resources (servers, nodes, clusters)

By default oms assumes:

all models are running on localhost
there are no limits on CPU cores or memory usage

Model run queue on local computer

You can create model run queue on your local computer by setting a limit on number of CPU cores available. To do it modify job.ini file in a job directory, for example:

[Common]
LocalCpu      = 8       ; localhost CPU cores limit, localhost limits are applied only to non-MPI jobs
LocalMemory   = 0       ; gigabytes, localhost memory limit, zero means no limits

You don't have to set memory limits until model run memory requirements are known.

CPU cores which are you limiting in job.ini does not need to be an actual cores. You can have 8 cores on your PC and set LocalCpu = 16 which allow 200% overload and may significantly slow down your local machine. Or if you set LocalCpu = 4 then your models would be able to use only half of actual cores.

LAN: front-end server and back-end cluster of servers

Example of local network (LAN) cluster:

small front-end server with 4 cores
4 back-end servers: cpc-1, cpc-2, cpc-3, cpc-4 with 16 cores each

[Common]
LocalCpu      = 4   ; localhost CPU cores limit, localhost limits are applied only to non-MPI jobs
LocalMemory   = 0   ; gigabytes, localhost memory limit, zero means no limits
MpiCpu        = 40  ; max MPI cpu cores available for each oms instance, zero means oms instances can use all cpu's available
MpiMemory     = 0   ; gigabytes, max MPI memory available for each oms instance, zero means oms instances can use all memory available
MpiMaxThreads = 8   ; max number of modelling threads per MPI process
MaxErrors     = 10  ; errors threshold for compute server or cluster

Servers   = cpc-1, cpc-2, cpc-3, cpc-4      ; computational servers or clusters

[cpc-1]
Cpu       = 16          ; default: 1 CPU core
Memory    = 0           ; zero means no limits

[cpc-2]
Cpu       = 16          ; default: 1 CPU core
Memory    = 0           ; zero means no limits

[cpc-3]
Cpu       = 16          ; default: 1 CPU core
Memory    = 0           ; zero means no limits

[cpc-4]
Cpu       = 16          ; default: 1 CPU core
Memory    = 0           ; zero means no limits

; OpenMPI hostfile (on Linux)
;
; cpm   slots=1 max_slots=1
; cpc-1 slots=2
; cpc-3 slots=4
;
[hostfile]
HostFileDir = models/log
HostName = @-HOST-@
CpuCores = @-CORES-@
RootLine = cpm slots=1 max_slots=1
HostLine = @-HOST-@ slots=@-CORES-@

; MS-MPI machinefile (on Windows with Microsoft MPI)
;
; cpm:1
; cpc-1:2
; cpc-3:4
;
; [hostfile]
; HostFileDir = models\log
; HostName = @-HOST-@
; CpuCores = @-CORES-@
; RootLine = cpm:1
; HostLine = @-HOST-@:@-CORES-@

Based on job.ini above oms will create MPI hostfile with back-end servers assignment for each particular model run.

In order to use that hostfile you should modify model run template(s) in openM++ etc/ directory. For example on Linux with openMPI:

{{/*
oms web-service:
  Template to run modelName_mpi executable on Linux using OpenMPI

It is not recommended to use root process for modelling

Oms web-service using template for exec.Command(exeName, Args...):
  - skip empty lines
  - substitute template arguments
  - first non-empty line is a name of executable to run
  - each other line is a command line argument for executable

Arguments of template:
  ModelName string            // model name
  ExeStem   string            // base part of model exe name, usually modelName
  Dir       string            // work directory to run the model
  BinDir    string            // bin directory where model exe is located
  MpiNp     int               // number of MPI processes
  HostFile  string            // if not empty then path to hostfile
  Args      []string          // model command line arguments
  Env       map[string]string // environment variables to run the model

Example of result:

  mpirun --hostfile host.ini --bind-to none --oversubscribe -wdir models/bin -x key=value ./modelName_mpi -OpenM.LogToFile false

*/}}

mpirun
--bind-to
none
--oversubscribe
{{with .HostFile}}
--hostfile
{{.}}
{{end}}
{{with .Dir}}
-wdir
{{.}}
{{end}}
{{range $key, $val := .Env}}
-x
{{$key}}={{$val}}
{{end}}
{{.BinDir}}/{{.ExeStem}}_mpi
{{range .Args}}
{{.}}
{{end}}

Note: If you are using OpenMPI then it is a good idea to have --oversubscribe --bind-to none as above in order to avoid MPI models run failure or performance degradation.

If you are using Microsoft MPI on Windows servers then modify etc\ model template file(s) to have it similar to:

{{/*
oms web-service:
  Template to run modelName_mpi.exe on Windows Microsoft MPI using machinefile

To use this template rename it into:
  mpi.ModelRun.template.txt

Oms web-service using template for exec.Command(exeName, Args...):
  - skip empty lines
  - substitute template arguments
  - first non-empty line is a name of executable to run
  - each other line is a command line argument for executable

Arguments of template:
  ModelName string            // model name
  ExeStem   string            // base part of model exe name, usually modelName
  Dir       string            // work directory to run the model
  BinDir    string            // bin directory where model exe is located
  DbPath    string            // absolute path to sqlite database file: models/bin/model.sqlite
  MpiNp     int               // number of MPI processes
  HostFile  string            // if not empty then path to hostfile
  Args      []string          // model command line arguments
  Env       map[string]string // environment variables to run the model

Example of result:
  mpiexec -machinefile hosts.ini -wdir models\bin -env key value ..\bin\modelName_mpi -OpenM.LogToFile false

*/}}

mpiexec
{{with .HostFile}}
-machinefile
{{.}}
{{end}}
{{with .Dir}}
-wdir
{{.}}
{{end}}
{{range $key, $val := .Env}}
-env
{{$key}}
{{$val}}
{{end}}
{{.BinDir}}\{{.ExeStem}}_mpi
{{range .Args}}
{{.}}
{{end}}

Cloud auto scaling: automatically start and stop servers

Use oms jobs control abilities to organize model runs queue and, if required, automatically scale up down cloud resources, e.g.: start and stop virtual machines or nodes.

For example, if you want to have two users: Alice and Bob who are running models then start oms as:

bin/oms -l localhost:4050 -oms.RootDir alice -oms.Name alice -ini oms.ini
bin/oms -l localhost:4060 -oms.RootDir bob   -oms.Name bob   -ini oms.ini

where content of oms.ini is:

[oms]
JobDir        = ../job
EtcDir        = ../etc
HomeDir       = models/home
AllowDownload = true
AllowUpload   = true
LogRequest    = true

[OpenM]
LogFilePath      = log/oms.log
LogToFile        = true
LogUseDailyStamp = true
LogToConsole     = false

Above assume following directory structure:

./    -> current directory
    bin/
        oms    -> oms web service executable, on Windows: `oms.exe`
        dbcopy -> dbcopy utility executable, on Windows: `dbcopy.exe`
    html/    -> web-UI directory with HTML, js, css, images...
             disk.ini  -> (optional) disk storage usage settings and shared models library
    etc/     -> config files directory, contain template(s) to run models
    alice/   -> user Alice "root" directory
        log/     -> recommended Alice's log files directory
        models/
              bin/  -> Alice's model.exe and model.sqlite directory
              log/  -> Alice's directory for models run log files
              doc/  -> models documentation directory
              home/ -> Alice's personal home directory
                  io/download  -> Alice's directory for download files
                  io/upload    -> Alice's directory to upload files
    bob/     -> user Bob "root" directory
        log/     -> recommended Bob's log files directory
        models/
              bin/  -> Bob's model.exe and model.sqlite directory
              log/  -> Bob's directory for models run log files
              doc/  -> models documentation directory
              home/ -> Bob's personal home directory
                  io/download  -> Bob's directory for download files
                  io/upload    -> Bob's directory to upload files
    job/  -> model run jobs control directory, it must be shared between all users
          job.ini   -> (optional) job control settings
          active/   -> active model run state files
          history/  -> model run history files
          past/     -> (optional) shadow copy of history folder, invisible to the end user
          queue/    -> model run queue files
          state/    -> jobs state and computational servers state files
               jobs.queue.paused      -> if such file exists then jobs queue is paused
               jobs.queue.all.paused  -> if such file exists then all jobs in all queues are paused

You don't have to follow that directory structure, it is flexible and can be customized through oms run options.

IMPORTANT: Job directory must be in a SHARED location and red-write accessible to all users who are using the same queue and the same computational resources (servers, nodes, clusters).

You don't need to create OS users, e.g. Alice and Bob does not need a login accounts on your server (cloud, Active Directory, etc.). All you need is to setup some authentication mechanism and reverse proxy which would allow Alice to access localhost:4050 and Bob localhost:4060 on your front-end. Actual OS user can have any name, e.g. oms:

sudo -u oms OM_ROOT=/shared/alice bash -c 'source ~/.bashrc; bin/oms -l localhost:4050 -oms.RootDir alice -oms.Name alice -ini oms.ini &'
sudo -u oms OM_ROOT=/shared/bob   bash -c 'source ~/.bashrc; bin/oms -l localhost:4060 -oms.RootDir bob   -oms.Name bob   -ini oms.ini &'

Cloud: IT admin user (global admin user)

Global admin special run option -oms.AdminAll allows IT admin to see all active users, history of model runs and resources usage (CPU and disk space):

bin/oms  -l localhost:4080 -ini oms.ini -oms.RootDir admin -oms.Name admin -oms.AdminAll

See the screenshot: Cloud administrator UI

Cloud: read-only user (Models Library)

Read-only oms instance allow to view the models but do not allow users to change the data, delete or upload anything or to run the models. It also can be used as Models Library shared storage from where users can copy model(s) into their own workspace:

bin/oms -l localhost:4048 -ini oms.ini -oms.RootDir model-lib -oms.Name model-lib -oms.Readonly

See UI screenshots at Copy Model from the Library.

Cloud disks usage

You may want to set the limits on disk space usage and enforce storage cleanup by users. It can be done through etc/disk.ini file (see example below).

If etc/disk.ini exists then oms web-service will monitor and report disk usage by user(s) and may set a limit on storage space. You can set a limit for individual user, group of users and grand total space limit on storage space used by all users. If user exceeding disk space quotas then she/he cannot run the model or upload files to cloud, only download is available. User can Cleanup Disk Space through UI.

Models library

Models Library configuration defined in [ModelLib] section of etc/disk.ini.

IMPORTANT: SrcRoot directory must be in a SHARED location and accessible for reading by all oms users who can copy models from the library.

Actual model files copy done by CopyCmd script, which you can find in etc directory of openM++ release archive. For example:

../etc/model-copy.sh models/bin/RiskPaths-3.0.0.0/RiskPaths.publish.lst ../model-lib /openmpp/alice RiskPaths-3.0.0.0

where arguments of CopyCmd script are:

models/bin/RiskPaths-3.0.0.0/RiskPaths.publish.lst : path to model publish list file (see below), if relative then must be relative to SrcRoot directory
../model-lib : Models Library path, if relative then must be relative to $OM_ROOT
/openmpp/alice : destination user workspace directory, if relative then must be relative to $OM_ROOT
RiskPaths-3.0.0.0 : model name or name-version

Model publish list file created by omc model compiler or just in text editor. It contains list of the files to be copied from Models Library directory into user workspace directory. For example:

$BIN_DIR/RiskPaths.exe
$BIN_DIR/RiskPaths.sqlite
$BIN_DIR/RiskPaths.ini
$BIN_DIR/RiskPaths.extra.json
$DOC_DIR/chap1-eng.pdf
$DOC_DIR/RiskPaths.doc.EN.html
$DOC_DIR/RiskPaths.doc.FR.html

$LOG_DIR/RiskPaths.2026_06_18_23_47_59_243.log
other/dir/extra.data.file

Last two lines in example above added into the RiskPaths.publish.lst file manually by typing it in text editor.

Following environment variables are used:

$OM_ROOT (optional) openM++ root path
$BIN_DIR default: models/bin sub-folder where model.exe and model.sqlite resides
$DOC_DIR default: models/doc models documentation sub-folder
$LOG_DIR default: models/log models log sub-folder

Example of disk.ini:

; Example of storage usage control settings
;   "user" term below means oms instance
;   "user name" is oms instance name, for example: "localhost_4040"
;
; if etc/disk.ini file exists then storage usage control is active

[ModelLib]
;
; URL of models library, usually read-only oms instance
;
Url = http://localhost:4048
;
; Source root of of models library, where models/bin models/log models/doc are located
;
SrcRoot = ../model-lib
;
; Model copy script:
;   copy model files from library $src_root to the user $dst_root
;   using model file list from ModelName.publish.lst
;
CopyCmd = ../etc/model-copy.sh
;
[Common]

; seconds, storage scan interval, if too small then default value used
;
ScanInterval  =   0

; GBytes, user storage quota, default: 0 (unlimited)
;
UserLimit     =   0

; GBytes, total storage quota for all users, default: 0 (unlimited)
;   if non-zero then it restricts the total storage size of all users
;
AllUsersLimit = 128

; Database cleanup script:
;   creates new model.sqlite database and copy model data
;
DbCleanup = ../etc/db-cleanup_linux.sh

; user groups can be created to simplify settings
;
Groups = Low, High, Others

[Low]
Users      = localhost_4040, bob, alice
UserLimit  = 2

[High]
Users      = king, boss, cheif
UserLimit  = 20

[king]
UserLimit  = 100 ; override storage settings for oms instance "king"

; "admin" is not a member of any group
;
[admin]
UserLimit  = 0 ; unlimited

To find additional information and examples please continue reading at Oms: Cloud and cluster setup.