Defining runs - GLOBIO4/GlobioModelPublic GitHub Wiki

Configuration files

If you want to do a model run you have to make a configuration file. Such a file defines which model parameters will be used for the run and which modules should be calculated. It's also possible to define your own modules and scenarios based on the standard calculations or define a batch run.

Configuration files are ASCII files which can be made with a simple text editor. It's good practice to use ".glo" as the file extension for your configuration files, but it 's not mandatory.

It is possible to save frequently used settings or scenarios in separate configuration files and include these file in your run file.

Configuration files can be very simple or fairly complex. You do not need knowledge of Python to make configuration files.

An example of a simple configuration file is:

# Set output directory.
Outdir = C:\Temp\Globio4\out_v1

# Set input raster.
RASTER Landuse = $InDir;\tif\glc2000.tif

# Set input lookup csv file.
FILE SettlementsLookup = $LookupDir;\Settlements.csv

# Set input/output raster.
RASTER Settlements = $OutDir;\Settlements.tif

# Include the base modules.

INCLUDE Globio_Modules.glo

#-------------------------------------------------------------------------------
BEGIN_RUN Calculations()

RUN_MODULE
Settlements($Extent,$Landuse,$SettlementsLookup,$Settlements)

END_RUN

#-------------------------------------------------------------------------------
#-------------------------------------------------------------------------------
RUN Calculations()

To start a model run, type the following command in a Windows command window:

globio4 <filename>

For example:

globio4 settlements_2016.glo

The master configuration file

GLOBIO 4 uses a master configuration file to define the predefined base impact modules and commonly used variables. This master configuration files resides in the directory \Config in the main directory and is named: Globio_Config.glo.

The master configuration file is maintained by the GLOBIO 4 administrator. Users can run the base modules and use the predefined variables and default settings. The values of these variables can be overridden in the users own configuration file.

The master configuration file is automatically included in each run.

When running under Linux instead of Globio_Config.glo the file Globio_ConfigLinux.glo will be included. If this file is not found the default file Globio_Config.glo is used.

Comments in configuration scripts

To add your own comment in a configuration file start a line with a #. All text on that line will be regarded as comment and will be ignored when parsing and looking for configuration settings.

Syntax:

# This is comment.

Using types and variables

To set model parameters in a configuration file variables can be used. Examples of variables are: InDir, OutDir, CellSize, SettlementsLookup.

Every variable has a type and the available types are:

  Obj          For internal use.
  Boolean      True or False.
  Integer      Positive or negative whole numbers with no decimal point.
  Float        Positive or negative numbers with a decimal point.
  String       Text string.
  CellSize     Cellsize for rasters.
  Extent       Regional extent represented as a list of minx, miny, maxx and
               maxy coordinates in decimal degrees separated by ",". 
               For example: 3.0,50.0,8.0,54.0
  Dir          A directory.
  File         A File.
  Raster       A raster dataset. Valid raster types are: ESRI Grid (read only)
               and TIF (read/write).
  RasterList   A list of raster datasets, separated by a "|". Valid raster
               types are: ESRI Grid (read only) and TIF (read/write).
  Vector       A vector dataset. Valid vector types are: FileGeodatabase (read
               only) and Shapefile (read/write).

By assigning a type to a variable (or model parameter) values of variables can be checked for validity. For example when a module has an integer parameter as input and a float value is given, an error message will be given. Or when a raster dataset is needed as input a check will be done if the dataset exists.

All listed types are build-in and can not be modified by a user.

To define a variable and set a value use the following syntax:

Syntax:

<type name> <var name> = <value>

Examples:

INTEGER BufferDistanceKM = 10
FLOAT WeightFactor = 0.666666666666
STRING FieldName = GP_RTP
DIR BaseDir = G:\Data\
FILE SettlementsLookup = G:\Data\Input\Settlements.csv
RASTER NaturalLanduse = T:\Temp\out_v3\NaturalLanduse.tif

Remarks:

  • It is recommended to use uppercase for type names. Configuration files will be more readable.
  • Type and variable names are not case-sensitive.
  • When defining a new variable always an (initial) value has to be set.

Values of previously defined variables can be changed. Just assign a new value.

Syntax:

<var name> = <value>

Examples:

BufferDistanceKM = 50

Using variables and variable concatenation

To use defined variables use the following syntax:

Syntax:

<type name> <var name2> = $<var name1>

<var name2> = $<var name1>

Examples:

INTEGER DistanceKM = $BufferDistanceKM

DIR TmpDir = $OutDir

Be aware type checks will be done. So you can not assign an INTEGER variable to a FLOAT variable or a DIR variable to a FILE variable.

Some types of variables can be concatenated. A variable value can be combined with text and assigned to an other variable. Also two or more variable values can be combined, even with text.

Syntax:

<type name> <var newname> = $<var name1>;<text>

<type name> <var newname> = $<var name1>;$<var name2>

<type name> <var newname> = $<var name1>;<text>$<var name2>

<var newname> = $<var name1>;<text>$<var name2>;<text>$<var name3>

Examples:

DIR InDir = $BaseInDir;\pbl

FILE SettlementsLookup = $LookupDir;\Settlements.csv

To reference an existing variable add a "$" to the variable name. When something succeeds the variable name use a ";" to mark the end of the name of the variable.

Remarks:

  • Assigning variables recursive is not allowed. So <var name> = $<var name> is not allowed.

The next overview shows which variable types can be assigned with a concatenated value and which types can be used in a concatenation.

  Name              Can assignd by concatenation    Can used in concatenation
  ----------------- ------------------------------- --------------------
  Obj               Yes                             No
  Boolean           No                              No
  Integer           No                              Yes
  Float             No                              No
  String            Yes                             Yes
  CellSize          No                              No
  Extent            No                              No
  Dir               Yes                             Yes
  File              Yes                             Yes
  Raster            Yes                             Yes
  RasterList        Yes                             Yes
  Vector            Yes                             Yes

Using constants

Additional to types and variables constants can be used. Constants are predefined "names" with a particular value. Each constant has a type. Constants are build-in and maintained by the GLOBIO 4 administrator. The value of a constant can not be changed by a user.

The following constants are defined:

  Name             Type       Python Value
  ---------------- ---------- -------------------------------
  True             Boolean    True
  False            Boolean    False
  UserName         String     Windows username
  UserTempDir      Dir        Windows user temp directory
  1deg             CellSize   1.0
  30min            CellSize   0.5
  5min             CellSize   0.0833333333333
  30sec            CellSize   0.00833333333333
  10sec            CellSize   0.00277777777778
  ValidCellSizes   String     1deg|30min|5min|30sec|10sec
  world            Extent     [-180.0, -90.0, 180.0, 90.0]
  wrld             Extent     [-180.0, -90.0, 180.0, 90.0]
  europe           Extent     [-25.0, 33.0, 45.0, 72.0]
  eu               Extent     [-25.0, 33.0, 45.0, 72.0]
  nl               Extent     [3.0, 50.0, 8.0, 54.0]

You can use these constants as follows:

SaveTmpData = True
EXTENT MapExtent = europe
CELLSIZE MapCellSize = 30sec

The advantage of using constants for CellSize or Extent for example, is that model parameters can easily been set with standardized values. Every run with a resolution of 30 seconds uses the same cellsize.

Remarks:

  • No $ sign is used for referencing a constant.
  • Names of constants are case-sensitive.
  • Names of constants can not be used in concatenation.

To list the actual predefined constants and their values, type the following command in a Windows command window:

globio4 -doc

Using runs, scenarios and modules

To define and execute a model run, a configuration file should contain at least a run definition and a run command. A run definition is an execution block with commands to be executed. The commands can be variable assignments or commands for running other blocks like scenarios, modules or calculations.

You can define a run as follows.

Syntax:

BEGIN_RUN <name>()
...
END_RUN

You can execute a run as follows:

Syntax:

RUN <name>()

Remarks:

  • Every configuration file should contain at least one run block and one run command.
  • In a run block no run commands can be executed.

Scenarios and modules are execution blocks which can be used to group related commands and make reusable blocks of commands. It not required to define and use scenarios and modules.

To define scenario and module blocks use the following syntax:

Syntax:

BEGIN_SCENARIO <name>(arguments)
...
END_SCENARIO

BEGIN_MODULE <name>(arguments)
...
END_MODULE

To run a scenario or module use the following syntax:

Syntax:

RUN_SCENARIO <name>(arguments)

RUN_MODULE <name>(arguments)

Remarks:

  • Run commands in a configuration file are executed in sequential order from top to down as listed in the file.

Passing arguments

To pass variable values to scenarios and modules when calling a scenario or module, arguments can be used. With arguments you have more control over which parameter values you use for your calculations.

To specify arguments for scenarios and modules use the following syntax:

Syntax:

BEGIN_SCENARIO <name>(IN <type name> <var name>, ..., OUT <type name> <var name>)

BEGIN_MODULE <name>(IN <type name> <var name>, ..., OUT <type name> <var name>)

Arguments are defined in the same way as variables with a type and a name. Additional the keyword IN or OUT must be used to specify if the argument is an input or an output parameter.

Input and output arguments are especially important for DIR, FILE, RASTER and VECTOR types. If one of these types is defined as input the specified directory or file must exist. If it is defined as output the specified directory or file should not already exist. Before a run starts validity checks are done on base of the defined input and output arguments.

Example:

BEGIN_MODULE Convert(IN FILE Csv,OUT FILE Txt)
...
END_MODULE

FILE CsvFile = C:\Temp\Lookup\NaturalLanduse.csv

FILE TxtFile = C:\Temp\Out_v2\NaturalLanduse.txt

RUN_MODULE Convert($CsvFile,$TxtFile)

Using optional arguments

When running scenarios and modules sometimes parameters are optional. To declare an optional variable use the value NONE.

Example:

FILE OutRegionAreasFileName = NONE

Global vs local variables

When using variables a difference can be made between global and local variables. Global variables are variables which can be used throughout the whole configuration file. Local variables are only valid in the block they are defined in. All predefined variables are global variables.

To define your own global variable place the definition outside a RUN, SCENARIO or MODULE block.

Example:

RASTER Landuse = $InDir;\tif\glc2000.tif

BEGIN_RUN Calculations2016()
...
END_RUN

You can use the Landuse variable in any run, scenario of module block in your configuration.

To define a local variable place the definition inside a RUN, SCENARIO or MODULE block. You can use this variable only in the block in which it is defined. Using it in an other block will result in an error.

Example:

BEGIN_SCENARIO Scenario2016()
    RASTER Landuse = $InDir;\tif\glc2000.tif
    ...
    RUN_MODULE CalcLanduseMSA($LandUse,...)
    ...
END_SCENARIO
BEGIN_SCENARIO Scenario2017()
    ...
    RUN_MODULE CalcLanduseMSA($LandUse,...)
    ...
END_SCENARIO

In this example the variable Landuse is a local variable of Scenario2016 and can only be used inside this block. The use of the variable Landuse in Scenario2017 is invalid and will result in an error.

The next example shows why it is preferred to use local variables and pass them as arguments and the use of global variables should be done with caution.

Example:

FLOAT BufferDistanceKM = 10
BEGIN_RUN Calculations2016()
    RUN_SCENARIO Scenario2016()
    RUN_SCENARIO Scenario2017()
END_RUN

BEGIN_SCENARIO Scenario2016()
    BufferDistanceKM = 15
    ...
    RUN_MODULE CalcHumanEncroachment($BufferDistanceKM,...)
    ...
END_SCENARIO
BEGIN_SCENARIO Scenario2017()
    ...
    RUN_MODULE CalcHumanEncroachment($BufferDistanceKM,...)
    ...
END_SCENARIO

The variable BufferDistanceKM is defined globally and initialized with the value 10. Inside the Scenario2016 block the value is changed and the same variable with the changed value is used in Scenario2017, This is probably not what you expect.

The next example shows the same calculation, but using local variables and arguments.

Example:

BEGIN_RUN Calculations2016()
    FLOAT BufferDistanceKM = 10
    RUN_SCENARIO Scenario2016($BufferDistanceKM,...)
    RUN_SCENARIO Scenario2017($BufferDistanceKM,...)
END_RUN
BEGIN_SCENARIO Scenario2016(IN FLOAT BufferDistanceKM)
    BufferDistanceKM = 15
    ...
    RUN_MODULE CalcHumanEncroachment($BufferDistanceKM,...)
    ...
END_SCENARIO
BEGIN_SCENARIO Scenario2017(IN FLOAT BufferDistanceKM)
    ...
    RUN_MODULE CalcHumanEncroachment($BufferDistanceKM,...)
    ...
END_SCENARIO

In this example Scenario2016 is executed with a buffer distance of 15 and Scenario2017 with a buffer distance of 10.

Using the GLOBIO calculations

Calculations are the execution blocks which performs the actual Python calculations. The Python code of the calculations are maintained by the GLOBIO 4 administrator(s) and can not be modified. A user can only run a calculation.

Names of calculations always starts with GLOBIO_.

To run a calculation use the following syntax:

Syntax:

GLOBIO_<name>(arguments)

In GLOBIO 4 the following calculations are available, among others:

GLOBIO_CalcClimateChangeMSA
GLOBIO_CalcDiscreteLanduseAllocation
GLOBIO_CalcDominantLanduse
GLOBIO_CalcHumanEncroachmentMSA
GLOBIO_CalcInfraDisturbanceMSA
GLOBIO_CalcInfraFragmentationMSA
GLOBIO_CalcLanduseBiomesMSA
GLOBIO_CalcLanduseMSA
GLOBIO_CalcNDepositionMSA
GLOBIO_CalcNaturalLanduse
GLOBIO_CalcSettlements
GLOBIO_CalcTerrestrialMSA
GLOBIO_CompareRasters

To list the actual calculations available and their arguments open in Windows a command window and type the following command:

globio4 -doc

Include configuration scripts

To reuse run, scenario and module block definitions you can save them in separate configuration files. These configuration files can be included in new configuration files.

To include a configuration file use the following syntax.

Syntax:

INCLUDE <file>

For file names without a path the file will be searched in the same directory as the current file. If no file is found the file will be searched in the GLOBIO 4 config directory. For files in other directories absolute and relative paths can be used.

Examples:

INCLUDE Modules_inc.glo
INCLUDE ../Include/Modules_inc.glo
INCLUDE C:\MyHome\Globio4\Includes\Modules_inc.glo

Remarks:

  • Defining global variables in included configuration files is not allowed.
  • The INCLUDE keyword can not be used inside RUN, SCENARIO and MODULE blocks.

Using lists

To do the same calculation for different years or with different input files you can use lists. With lists you can assign subsequent values to a variable and execute the enclosing list block for each value.

For defining lists use the following syntax:

Syntax:

LIST <type name> <var name> = <value1>|<value2>|<value3>|...
...
END_LIST

Example:

LIST INTEGER Year= 2000|2001
    RUN_MODULE CalcLanduse($Year,...)
END_LIST

The variable YEAR is assigned subsequently to the values 2000 and 2001 and for each value the module CalcLanduse is executed.

Lists can also be nested.

Example:

BEGIN_SCENARIO Scenario3()
    DIR TestDir = C:\Test\
    LIST INTEGER Year = 2000|2001
        LIST FILE Landuse= landuseA|landuseB
            RUN_MODULE CalcNDep($TestDir,$Landuse,$Year)
        END_LIST
    END_LIST
END_SCENARIO

Remarks:

  • Lists can only be used inside RUN, SCENARIO and MODULE blocks.

Special function: MSG

To show your own messages during a model run you can use the MSG function.

Syntax:

MSG(<text>)
MSG(<var name>)

Examples:

MSG(This scenario is a special variant of scenario 2016.)
MSG(Using landuse: $Landuse)

Logging

To ensure the reproducibility of the calculated results by default all relevant run information is saved in a log file. The information includes the name of the executed modules, their arguments and start time and duration of the run.

The logfile is saved in the output directory, which is the value of variable OutDir, in a file with the name "_globio4.log".

It is possible to disable logging to a file by adding the following line to your configuration file.

LogToFile = False

The next example shows the logging information of a run.

================================================================================
# GLOBIO 4.0, apr 2016                                           (Framework 1.0)
#
# Execution start: 20160408 12:17:43
================================================================================
Loading file Settlements.glo
Loading file P:\Project\Globio4\Config\Globio_Modules.glo

--------------------------------------------------------------------------------
- Running GLOBIO_CalcSettlements
--------------------------------------------------------------------------------
Arguments:
  [-180.0, -90.0, 180.0, 90.0]
  P:\Project\Globio4\data\pbl\tif\glc2000.tif
  P:\Project\Globio4\data\Lookup\Settlements.csv
  G:\Data\out_v5\Settlements.tif
Starting run...
Using cellsize: 0.00833333333333
Reading landuse raster...
Reading lookup...
Reclassing...
Writing G:\Data\out_v5\Settlements.tif...
--------------------------------------------------------------------------------
- Execution time: 1 min 5 sec
--------------------------------------------------------------------------------

================================================================================
# Execution successfully ended.
# Total execution time: 1 min 5 sec
================================================================================```

Overwrite output

Existing output datasets can be overwritten by adding the following line to your configuration file.

OverwriteOutput = True

By default existing output is not overwritten.

Creating the output directory

Automatically create the full paths for output datasets by adding the following line to your configuration file.

CreateOutDir = True

By default non-existing output paths are not created.

Number of cores

The number of cores used for parallel calculations can be set by adding the following line to your configuration file.

NumberOfCores = <value>

With:

value >= 1 This number of cores will be used.

value = 0 All number of available cores will be used.

value < 0 All number of available cores will be used minus the specified number.

By default all number of available cores will be used.

Example:

NumberOfCores = -1

In this example all available cores will be used during the execution of the script except one for system/administrative processes.

Saving temporary files

During some impact calculations temporary datasets are created. By default these datasets are not saved to disk. It is possible to save these datasets to disk by adding the following line to your configuration file.

SaveTmpData = True

You can use those temporary files to check your data and the calculations.

Error messages

If you start a run the configuration file is loaded and validity checks will be done. During these checks error messages can be shown. An error message consists of an error description, the name of the file in which the error occurred, and a line number in that file.

    Error: <error description> - <script filename>:<line number>

Example:

Error: Invalid number of arguments in call. - Run_All.glo:79

Styling guidelines

When making a configuration file the following styling guidelines can be used to write your run definitions.

  • comment your scripts;
  • write names of types in uppercase: for example INTEGER;
  • write begin, end and run keywords in uppercase: for example BEGIN_MODULE, END_MODULE, RUN_MODULE;
  • write the keywords IN and OUT in uppercase;
  • in arguments define IN arguments first, then OUT arguments;
  • use #------------------------- lines to mark the start of a begin-block;
⚠️ **GitHub.com Fallback** ⚠️