Exec_ec - ObjectVision/GeoDMS GitHub Wiki

File, Folder and Read functions exec(ute)_ec(errorcode)

• EXEC_EC(command line)
• EXEC_EC(command line, current folder path)
• EXEC_EC(application name, command line, current folder path)

executes the command line argument and returns its ExitCode.

It does so by calling CreateProcessA(application name, command line, NULL, NULL, TRUE, 0, NULL, NULL, zeroes, zeroes) where application name is NULL when not specified, in which case the application to start is derived from the command line. Once started, command line is what the child process will see when calling GetCommandLine(). See also: Environment.cpp.

When no current folder path is specified, the current folder is inherited from the GeoDms process, which usually is set to the project folder of the last loaded configuration.

You can use such a result in the construction of a storage name of a data source which guarantees that it will only be known after completion of that process.

• command with string value type

This example shows how to use the exec_ec function to make a list of files in a folder and store the resulting list in a text file, that can be used later in the process to read all files from the folder.

The CanGenerate parameter can be used in your expression to process all files, making sure the list of files or an error code is generated first.

container folderinfo
{
  container impl
  {
     parameter<string> FileNameDirInfo := '%LocalDataProjDir%/dirinfo_' + date +'.str';
     parameter<string> DirCmdOrg       := expand(., 'Dir '+ XmlDir +'/*.xml > ' + FileNameDirInfo);
     parameter<string> DirCmd          := replace(DirCmdOrg, '/', '\\') + ' /B';
   }

   parameter<uint32> writeFileList     := 
      EXEC_EC(expand(., '%env:ComSpec%'), '/c ' + impl/DirCmd, expand(., '%LocalDataProjDir%'));
   parameter<bool>   CanGenerate       := writeFileList== 0;
}

container wait_for_exec_ec
{
    parameter<uint32> exitcode := EXEC_EC(
        "python",
        "C:/path/to/script.py --output C:/path/to/output.csv"
    );
    unit<uint32> output
        StorageName   = = "C:/path/to/output" + string(exitcode - exitcode) + ".csv"
        StorageType   = "gdal.vect"
        StorageReadOnly = "True"
    {
        attribute<string> indicator_A;
    }
}

The term string(exitcode - exitcode) always evaluates to the string "0", so it does not change the resulting file name. Its only purpose is to make the StorageName depend on exitcode: because the storage name can only be determined after exitcode is known, the GeoDMS first runs the exec_ec process (the Python script) and only then opens output and reads it. This guarantees the script has finished before its output is read.

Since GeoDMS 20.16.0 the GeoDMS gives the child one pipe for both its stdout and its stderr and copies what arrives into the message log, line by line, as exec: <line> under the commands message category. Before that release the child simply inherited the GeoDms std handles: under GeoDMS GUI, which has no console, a failing Rscript or python left nothing behind but its ExitCode, which made a broken script very hard to diagnose.

Two details worth knowing:

• A redirection inside the command itself still wins, as it always did. In cmd /c dir *.shp > dirinfo.str the listing goes to the file and only what is not redirected reaches the message log.
• At most 1 MB of child output is copied into the log. Beyond that the output is still read - not reading it would block the child once the pipe buffer fills - but no longer reported, and a warning says so once.

Also since GeoDMS 20.16.0, a child that runs for a long time reports exec: still waiting for <program> after <n> seconds every ten seconds, so that a multi-minute model run is not a silent black box. Note that exec_ec still occupies the main thread for the duration of the child: the operator has external effects, which keeps it off the worker pool, so the GeoDms user interface does not repaint while a child runs.

exec_ec runs when its result is demanded, and what consumes the result decides the moment:

• Consumed by meta-scripting - an indirect StorageName as in the constructions above - the child is started while the GeoDMS is still establishing meta-information, before any data is calculated. The storage name cannot be established until exitcode is known, so the child runs first and the file is read afterwards.
• Consumed as an ordinary supplier - via ExplicitSuppliers on the item that must wait for it - the child runs at calculation time, when the demand reaches it through the supplier chain. Since GeoDMS 20.16.0 this is a fully supported ordering for reading storages too; see the next sections.

In both cases the operator's calc_requires_metainfo policy keeps the child on the main thread for the duration of the run.

The meta-scripting form sets a hard limit in the other direction.

When exec_ec runs at meta-information time - that is, when its result is consumed by meta-scripting such as an indirect StorageName - a storage that the GeoDMS itself writes cannot be forced to happen before it within a single run, because writing happens when data is calculated, which is after meta-information. Each of the following was tried and does not force the write first:

• PropValue(input, 'StorageName'). PropValue only reads the property text; its first argument carries the calc_never argument policy, so the named item is never calculated. An earlier version of this page claimed that this construction forces the input to be written first - that was wrong, and a configuration relying on it runs the child against a missing or stale input file.
• ExplicitSuppliers on the exec_ec parameter, naming either the write unit or one of its attributes.
• A data dependency such as sum(input/id) woven into the command line.

ExplicitSuppliers is the right tool for the opposite order: an item that may only be read after an exec_ec has run. This is how a revision-info file is picked up in the 2UP configuration:

container SvnImpl
{
   parameter<string> RevisionInfoCmd := replace(expand(., 'SubWCRev %projDir% > %projDir%/SubWCRevData.str'), '/', '\\');
   container Writer := EXEC_EC(expand(., '%env:ComSpec%'), '/c ' + RevisionInfoCmd, expand(., '%projdir%'));
}
...
parameter<string> RevisionInfo
:  StorageName       = "%projDir%/SubWCRevData.str"
,  ExplicitSuppliers = "SvnImpl/Writer";

Since GeoDMS 20.16.0 a full write → run → read round trip can run inside a single GeoDmsRun call, with every step an ordinary supplier of the next. The reader declares its layout, defers all file access past meta-information, and names the exec_ec result as its explicit supplier:

// step 1: written by the GeoDMS; forced before the exec by ITS ExplicitSuppliers
container input : StorageName = "= Dir + '/input.parquet'", StorageType = "gdalwrite.vect"
{
   attribute<uint32>  id          (src/objecten) := src/objecten/id;
   attribute<float32> oppervlakte (src/objecten) := src/objecten/oppervlakte;
}

// step 2: the script; the write happens first because input is its explicit supplier
container run
{
   parameter<uint32> exitcode := EXEC_EC("python"
      ,  '"%LocalDataProjDir%/Python/script.py" "' + Dir + '/input.parquet"'
      ,  Dir), ExplicitSuppliers = "../input";
}

// step 3: the reader; declared layout, no file access at meta, read waits for the exec
unit<uint32> output := src/objecten
:  StorageName       = "= Dir + '/output.parquet'"   // filename only: no exec reference
,  StorageType       = "gdal.vect"
,  StorageReadOnly   = "True"
,  SyncMode          = "None"                        // defer ALL file access past meta
,  ExplicitSuppliers = "run/exitcode"
{
   attribute<float32> result: ExplicitSuppliers = "run/exitcode";
}

The elements that make this work, each indispensable:

• SyncMode = "None" on the reader keeps the meta-phase free of the file: no open, no existence check, so the configuration loads while the file does not exist yet. This requires the domain and all attributes to be declared - with SyncMode = "None" nothing is synchronised from the storage, and a stored array whose length does not match its declared domain is refused at read time.
• A filename-only indirect StorageName - it may compose paths and parameters, but must not reference the exec_ec result, or the child is dragged back to meta-information time.
• ExplicitSuppliers on the reader unit and on each declared attribute, naming the exec_ec result. Each attribute read then waits for the child to complete. No PhaseContainer fence is needed for this: the ExplicitSuppliers become prerequisites of the read itself, which holds with the parallel scheduler on - the very case in which an unfenced read used to run ahead of the child.
• The exec_ec item's own ExplicitSuppliers naming the written input: at calculation time this DOES order the GeoDMS write before the child - the limitation of the previous section applies to the meta-time form only.

On releases before 20.16.0 this construction silently fails: the read of a stored item was scheduled without honouring its ExplicitSuppliers, so the read races ahead of the still-running child and finds a missing or half-written file. On those releases, use the meta-scripting patterns above or the batch-file round trip below.

Why go through this trouble: in a model that steps through years (or other chained periods) with one external call per step, the meta-scripting form runs each call in its own isolated update wave, and everything computed for earlier steps is released between waves. Step k then re-derives the state of all k−1 previous steps: computation time per step grows linearly and the total quadratically. With the supplier form the whole chain is under interest before anything substantial is calculated, each step's state stays alive exactly until the next step has consumed it, and the total is linear. Measured on a 13-step national model: constant 13-16 s per step instead of 79-188 s growing, total 6 minutes instead of 33, peak memory 28 GB instead of 68 GB.

On releases before 20.16.0 - or when the extra robustness of process isolation is worth two extra process starts - a full round trip write → run → read can be driven from outside the GeoDMS, with one GeoDmsRun call per step. Each call is a separate process, so step n+1 starts only after step n has finished and closed its files - which settles the synchronisation question completely, and is a good deal easier to debug than one long dependency chain.

RunModel.cmd:

@echo off
set GEODMS="C:\Program Files\ObjectVision\GeoDms20.16.0\GeoDmsRun.exe"
set CFG=%~dp0cfg\main.dms

rem 1. let the GeoDMS calculate and write the input file
%GEODMS% %CFG% /run_model/input
if errorlevel 1 exit /b 1

rem 2. run the external model on it
%GEODMS% %CFG% /run_model/exitcode
if errorlevel 1 exit /b 1

rem 3. calculate whatever consumes the model output
%GEODMS% %CFG% /run_model/result

with a configuration along these lines:

container run_model
{
   parameter<string> Dir := expand(., '%LocalDataProjDir%/temp');

   // step 1: written by the GeoDMS when /run_model/input is requested
   unit<uint32> input := src/objecten
   ,  StorageName     = "= Dir + '/input.parquet'"
   ,  StorageType     = "gdalwrite.vect"
   ,  StorageReadOnly = "False"
   {
      attribute<uint32>  id          := src/objecten/id;
      attribute<float32> oppervlakte := src/objecten/oppervlakte;
   }

   // step 2: run the script on it
   parameter<uint32> exitcode := EXEC_EC("python"
      ,  '"%LocalDataProjDir%/Python/script.py" "' + Dir + '/input.parquet"'
      ,  Dir);

   // step 3: read the output; the ExitCode in the StorageName makes the read wait for step 2
   unit<uint32> output
   :  StorageName     = "= exitcode == 0 ? Dir + '/output.parquet' : ''"
   ,  StorageType     = "gdal.vect"
   ,  StorageReadOnly = "True"
   {
      attribute<float32> result;
   }

   parameter<float32> total := sum(output/result);
}

Note that steps 2 and 3 may share one call, as in the example above: within that call the ExitCode ordering does hold, since both happen at meta-information time. It is only the GeoDMS write in step 1 that has to be a call of its own.

See parquet for the exchange format and for how to relate the read output domain to an already configured domain, R for running an R model this way, and Julia for a production pipeline that does exactly this across many study areas.

• exec
• execDll
• ExplicitSuppliers
• parquet
• R
• Julia