7. Running the models - NOC-MSM/SRIL34 GitHub Wiki
To run the model, first copy you NEMO and XIOS executables in the directory you want to run the model from, for eg:
cp /work/n01/n01/<user>/<path_to_nemo_compile>/NEMO_4.0.4/cfgs/<configuration_name>/BLD/bin/nemo.exe .
cp /work/n01/n01/<user>/<path_to_xios_compile>/xios-2.5/bin/xios_server.exe .
Now rename the NEMO executable, as archer2 has issues reading the *.exe appendix. IMPORTANT: only rename the executable that you copied in your running directory!
cd /<run_directory>
mv nemo.exe nemo
To run the model you must have your executables nemo and xios.exe, the domain_cfg.nc, the coordinates.nc , in your running directory, together with the *.xml file (that should come up when you compile nemo), your namelist_ref and your namelist_cfg, for eg :
-rwxr-xr-x 1 jrule n01 109310 Nov 4 2021 namelist_ref
-rwxr-xr-x 1 jrule n01 1702 Nov 4 2021 context_nemo.xml
-rwxr-xr-x 1 jrule n01 12948 Nov 4 2021 domain_def_nemo.xml
-rwxr-xr-x 1 jrule n01 110461 Nov 4 2021 field_def_nemo-oce.xml
-rwxr-xr-x 1 jrule n01 4420 Nov 4 2021 file_def_nemo-oce.xml
-rwxr-xr-x 1 jrule n01 6922 Nov 4 2021 grid_def_nemo.xml
-rwxr-xr-x 1 jrule n01 1156 Nov 4 2021 iodef.xml
-rwxr-xr-x 1 jrule n01 22089760 Nov 3 2021 nemo
-rwxr-xr-x 1 jrule n01 12331336 Nov 3 2021 xios_server.exe
-rwxr-xr-x 1 jrule n01 20006632 Nov 2 2021 coordinates.nc
-rwxr-xr-x 1 jrule n01 642521292 Nov 2 2021 domain_cfg.nc
-rwxr-xr-x 1 jrule n01 62306 Dec 22 2021 namelist_cfg
Amend the namelist_cfg with the paths to your boundary conditions, and restart files (or initial conditions). Also, set the parameters of the run.
Now all is set-up, you can run the model using a slurm script. To create the slurm script, see archer's instruction. An example of what you might run on your command line (changing the n01_SANH to your account name) is:
/work/n01/shared/nemo/mkslurm_hetjob -S 2 -s 8 -m 2 -C 75 -g 4 -N 128 -t 00:10:00 -a n01-SANH -j TEST > TEST.slurm
to run it use:
sbatch TEST.slurm
To generate the passive tracer release scenarios use the script create_scenario.python. This will generate a NetCDF file with daily information on the tracer release (in this example urea) which will be read in by the model.
To generate your scenario, activate your conda environment first (eg. nrct_env_py3). On livljobs, use:
module load anaconda
conda activate nrct_env_py3
Amend the create_scenario.python to set up your scenario. The script is set to generate a file starting from 2021-01-01 00:00:00 UTC. From this date, you choose the day in which the tracers start being release day0s, the duration of the leak in number of days leak_durations, the amount of tracer being released total_tonnes and the conversion factor to from tonnes to mmol (eg. urea). Also choose the coordinates of the tracer release location (based on the SRIL34 domain), and chose the output file name for you release scenario.
NOTE: the conversion factor will change depending on the molar mass of the chemical you are simulating. For urea, this is 60.06g/mol)
ntracers=1 # number of tracers to generate
day0s=array([144],dtype=int) # first day of leak for each tracer (144=25 May)
leak_durations=array([1],dtype=int) # duration of the leak for each tracer in days
total_tonnes=array([1100.]) # total mass in tonnes for each of the tracer
conversion_factors=array([16650016.65]) # tonnes to mmol (=1e9/60.06) - Urea
Yindex=array([153],dtype=int) # Lat index per each tracer leak
Xindex=array([102],dtype=int) # Lon index per each tracer leak
name='Scenario_urea1100_leak1day.nc'
Once your scenario is ready, run the script with:
python create_scenario.python
You should now have a 365 days scenarios file. Upload this in your running directory (on Archer2). The file is read in by the model through the fabm_input.nml; make sure you have the right Scenario name here before running the model. Also, as your file is daily, make sure the frequency is of 24 hours.
! ! file name ! frequency (hours) ! variable ! time interp. ! clim ! 'yearly'/ ! weights ! rotation ! land/sea mask !
! ! ! (if <0 months) ! name ! (logical) ! (T/F) ! 'monthly' ! filename ! pairing ! filename !
&variable
name='T1_flux/flux'
sn= 'Scenario_urea1100_leak1day.nc',24,'T1_flux', .false., .true., 'yearly', '', '', ''
to simulate tracer releases that occur in less than a daily, you will need to generate and hourly tracer scenario release. To do this use the script create_scenario_hourly.python. This is essentially the same as create_scenario.python, but with some changes to express everything in hours, rather than days. This means that when you change the script to set up your scenario, you will need to set the start time of you tracer release, and the duration of the leak in hours rather than days. eg, to set a scenario where the passive tracers are released in 6 hours:
ntracers=1 # number of tracers to generate
day0s=array([144*24],dtype=int) # first day of leak for each tracer (144=25 May)
leak_durations=array([6],dtype=int) # duration of the leak for each tracer in hours
total_tonnes=array([1100.]) # total mass in tonnes for each of the tracer
conversion_factors=array([16650016.65]) # tonnes to mmol (=1e9/60.06) - Urea
Yindex=array([153],dtype=int) # Lat index per each tracer leak
Xindex=array([102],dtype=int) # Lon index per each tracer leak
name='Scenario_urea1100_leak6hour.nc'
Also, before running the model, make sure that you amend frequency in the fabm_input.nml to read in the scenario in hours rather than days.
! ! file name ! frequency (hours) ! variable ! time interp. ! clim ! 'yearly'/ ! weights ! rotation ! land/sea mask !
! ! ! (if <0 months) ! name ! (logical) ! (T/F) ! 'monthly' ! filename ! pairing ! filename !
&variable
name='T1_flux/flux'
sn= 'Scenario_urea1100_leak6hour.nc',1,'T1_flux', .false., .true., 'yearly', '', '', ''
To run the SRIL34-FABM coupled configuration, the first step are the same as for the physics only (described above).
You need to copy the nemo and xios executable in your running directory, and change the physics namelist_cfg in the same way you would change the physics only model.
You will need to make sure that your SANH_FABM configuration is using the right keys. To check, open the /cfgs/SANH_FABM/cpp_SANH_FABM.fcm file (saved where you compiled nemo) with vi cpp_SANH_FABM.fcm. They keys should be:
bld::tool::fppkeys key_mpp_mpi key_vectopt_loop key_nosignedzero key_iomput key_top key_fabm
if this is not the case, add them to the file.
You will need extra *.xml files your directory compared to the phisics only. This is a list of what your .xml should look like:
-rwxr-xr-x 1 jrule n01 1912 Apr 28 15:49 context_nemo.xml
-rwxr-xr-x 1 jrule n01 12948 Apr 28 15:49 domain_def_nemo.xml
-rwxr-xr-x 1 jrule n01 309 May 4 13:52 field_def_custom.xml
-rwxr-xr-x 1 jrule n01 3787 Sep 1 12:46 field_def_fabm.xml
-rwxr-xr-x 1 jrule n01 110461 Apr 28 15:49 field_def_nemo-oce.xml
-rwxr-xr-x 1 jrule n01 29695 Apr 28 15:49 field_def_nemo-pisces.xml
-rwxr-xr-x 1 jrule n01 5319 Jul 22 10:46 file_def_nemo_1d_setting.xml
-rwxr-xr-x 1 jrule n01 5319 Jul 22 10:50 file_def_nemo_1h_setting.xml
-rwxr-xr-x 1 jrule n01 5319 Aug 2 15:18 file_def_nemo.xml
-rwxr-xr-x 1 jrule n01 6922 Apr 28 15:49 grid_def_nemo.xml
-rwxr-xr-x 1 jrule n01 1156 May 4 13:52 iodef.xml
you will also need a namelist_top_ref and namelist_top_cfg to control the tracers. Before running, set the run information in the namelist_top_cfg.
The tracer scenarios are not defined in the namelist file. When running the model will automatically look for a NetCDF file defining the scenario, so make sure you also have this file (eg: Scenario_urea1100_leak1day.nc) in the running directory.
With all files in place you can run the model using a slurm script, eg:
sbatch TEST.slurm