Overview

Documentation on routine (not in OptClim) running with MITgcm on Archer2 is at https://docs.archer2.ac.uk/research-software/mitgcm/.

An example of ECCOv4 being used in an illustrative study is:

• /work/n02/shared/mjmn02/OptClim/optclim2A/studies/ecco/Exampdfols_TestRunKeep/
• with the json file that defines the study being /work/n02/shared/mjmn02/OptClim/optclim2A/ModelOptimisation/ECCOv4/example_ecco_dfols.json

This uses 2 parameters just to illustrate the methods. That it is merely illustrative can be seen when you notice one is gravity! MITgcm models are usually run in a directory called "run" with links to all input data and the executable itself, and the usual SLURM script to run the model.

The "base" model is the built but not yet run model upon which all OptClim runs are based - after copying and modification to parameters in namelists.

Each run is in its own directory under the study directory. In the example on Archer2 the study top directory is as given above, and within it:

• the json defines the study, including the prefix zd for each run directory
• the runs are in subdirectories zd001,2......
• The "base" model which is copied for each instance of a model run under optClim will include the usual SLURM script.
• results from each run are in zd???/observables.json
• the directory jobOutput has the SLURM output files:
  • prefix PP: files from the array of jobs that create the simulated observables
  • prefix RE: 2nd and subsequent run of the optimiser in a job that initiates the next set of runs.

Preparing a Study

In one step you will need support.

• Create a directory for all your OptClim work.
• In that, create a script including the PATH to the required python:

 # for OptClim:
export OPTCLIMTOP=/where you cloned github/ModelOptimisation
export PATH=/work/n02/shared/mjmn02/sw/conda/opt_1/bin/:$OPTCLIMTOP/tools:$PATH
export PYTHONPATH=$OPTCLIMTOP:$OPTCLIMTOP/OptClimVn2:$PYTHONPATH

• code the generation of simulated observables
  • the example study uses ModelOptimisation/ECCOv4/postRun/simobs_json_1.sh to simply extract data from STDOUT.0000. This permits testing of the system.
• Set up the study directory an defining JSON file
  • make an empty top directory in which the study and all its runs will be held
  • Copy a JSON file that defines a study into the study's new top directory (link to generic documentation on this) and edit it so that it defines
    • the parameters to be tuned
    • the simulated observations and their targets
    • script that runs the calculation of simulated observables.
• The code in MITgcm.py currently has all supported parameter names hardcoded, so for first time a parameter is used in a study this code must be amended. See below for guidance.
• Edit the slurm script that runs the model
  • After the srun, insert code to check that the model completed successfully and if so the insert a line that runs a script called optclim_finished.
    • That script releases a held job that then calculates the simulated observables. It is generated and inserted in the run directory by OptClim.
  • The SLURM script with the added lines needs to be (the srun line may change according to the model):

srun --distribution=block:block --hint=nomultithread ./mitgcmuv

# check if job completed ok....
$OPTCLIMTOP/ECCOv4/complete_ok.sh 
retVal=$?
if [  $retVal -eq 0 ]; then
	 bash ./optclim_finished   # releases the array job corresponding to this run, the job that calculated simobs
else
    echo test complete_ok failed - not releasing the job to make simulated observations
fi

• If a MITgcm-based model chains multiple jobs for one model then teh SLURM script is more complex (for the routine model runs) and so locate the above lines so they are only run on completion of the chain.
• Run the study -- on the login node, after checking your environment is setup for both OptClim and MITgcm, cd to the parent of the study directory

runAlgorithm.py -v link_ecco1.json  > run.trace 2>&1 &

Notes for developers

pytest

After setting up the OPTCLIMTOP environment variable

cd $OPTCLIMTOP/ModelOptimisation
pytest -s OptClimVn2/test_MITgcm.py

The option -s makes visible output from print statements.

The intention is to test the relevant methods in MITgcm.py and its parent class ModelSimulation. This code is based on OptClim2 which had functions for the model HadCM3 that are not needed here.

Adding namelist parameters

How to add a new parameter when a namelist needs to be changed. There are two cases:
1) Simple: one variable in the framework matches to one namelist variable which has one value  
  Then use simpleNameList(var_name) which does most of the work for you.
    
2) Complex: one variable in the framework matches to multiple values in the namelist. Either an array for a variable
  or multiple variables.  
   In that case you need to write a function which takes a variable with a single default value and the following arguments:
      namelist (default False) which will return the namelist information, inverse (default False) which will query the 
        input namelist information and return the frame variable value.  To create the namelist info use
         _namedTupClass function which is inherited from ModelSimulation. 
         You need to specify the var, namelist name and file.  These named tuples are keys into a dict which you 
          use for the inverse function or need to create the dict for the forward function. See code below for 
          examples. 
  Then register the function with the registerMetaFn method. This method ties framework variable names to the function.