Link to status of mizuRoute

Notes from (roughly) weekly meeting.

From Oct 1st 2023 till the end of 2023, meeting is bi-weekly.

I am willing to contribute for MOSART (and mizuRoute) diagnostics. I have some notebooks to building on

First, to get PE layout, type ./pelayout under case directory. To change PE layout, consider hybrid PE layout for ROF component (using MPI tasks and threads). Make sure that total processors = NTASKS * NTHRDS
e.g., if we have 10 nodes, meaning that 1280 cores are available, these are acceptable 10 MPI tasks (NTASKS=10) and 128 threads (NTHRDS=128)
40 MPI tasks (NTASKS=40) and 32 threads (NTHRDS=10)
128 MPI tasks (NTASKS=128) and 10 threads (NTHRDS=10) \

You will have to turn on ESMF_AWARE_THREADING
./xmlchange ESMF_AWARE_THREADING=True

Reminder to change PE layout.
to change NTASKS for ROF,
./xmlchange ROF_NTASKS=10
to change NTHRDS for ROF,
./xmlchange ROF_NTHRDS=10 \

sequential vs. concurrent
You can change ROOTPE (starting core ID) for each component. For example, to change ROOTPE for ROF,
./xmlchange ROOTPE=3968 \

In the PE layout below, ATM and ROF have their own task sets. ATM use 0-127 cores (128 tasks) and CPL/LND/ICE/OCN/GLC/WAV have 128-3967 cores (3840) and ROF use 3968-5247 cores (1280 tasks)

Comp  NTASKS  NTHRDS  ROOTPE PSTRIDE
CPL :   3840/     1;    128      1
ATM :    128/     1;      0      1
LND :   3840/     1;    128      1
ICE :   3840/     1;    128      1
OCN :   3840/     1;    128      1
ROF :   1280/     1;   3968      1
GLC :   3840/     1;    128      1
WAV :   3840/     1;    128      1
ESP :      1/     1;      0      1

To performance test, use ./create_test under cime/scripts directory

e.g.,
./create_test PFS.f09_mg17,I2000Clm60Sp.derecho_intel.clm-default

Updates:

• Naoki is working on a floodplain implementation currently.
• Naoki may start working on tracers
• Naoki is also looking at future prediction of lakes as a project (especially endorheic lakes)
• After dust emissions, Erik should be able to get back to mizuRoute.
• First priority for Erik is to get mizuRoute working again/tested in CESM tags
• And also to have mizuRoute as part of CTSM checkouts
• Next priority for Erik is passing lake fields

Priorties evaluation:

1. floodplain
2. tracers
3. managed reservoirs?
4. Visualization of top 50 rivers and output to each ocean?
5. lakes in CTSM
6. Dataset creation for Paleo work?
7. Coupling to MOM See Aug-2-2023
8. Coupling to hillslope model in CTSM?

Q1: Do CESM use end or start of time step (e.g., currTime, startTime, stopTime etc.)? this is where they are defined in nuopc/rof_comp_nuopc.F90

currTime: current time (for initialization phase, it is equal to startTime for start up run or restart time for restart run) startTime: simulation start time (it is the same for start up run or restart run) stopTime: simulation end time

 432     !----------------------
 433     ! Initialize time managers in mizuRoute
 434     !----------------------
 435 
 436     ! Get clock properties
 437     call ESMF_ClockGet( clock, &
 438          currTime=currTime, startTime=startTime, stopTime=stopTime, refTime=RefTime, &
 439          timeStep=timeStep, rc=rc)
 440     if (ChkErr(rc,__LINE__,u_FILE_u)) return
 441 
 442     ! Get coupling interval in day
 443     call ESMF_TimeIntervalGet( timeStep, d=coupling_period, rc=rc)
 444     if (ChkErr(rc,__LINE__,u_FILE_u)) return
 445 
 446     ! get ymd integer and string for start time, end time and reference time
 447     ! for continuous run, start time == current time
 448     call ESMF_TimeGet( currTime, yy=yy, mm=mm, dd=dd, s=curr_tod, rc=rc )
 449     if (ChkErr(rc,__LINE__,u_FILE_u)) return
 450     call shr_cal_ymd2date(yy,mm,dd,curr_ymd)
 451 
 452     call ESMF_TimeGet( startTime, yy=yy, mm=mm, dd=dd, s=start_tod, rc=rc )
 453     if (ChkErr(rc,__LINE__,u_FILE_u)) return
 454     call shr_cal_ymd2date(yy,mm,dd,start_ymd)
 455 
 456     ! Always have simulation start at current time
 457     call shr_timeStr( currTime, simStart )
 458 
 459     call ESMF_TimeGet( stopTime, yy=yy, mm=mm, dd=dd, s=stop_tod, rc=rc )
 460     if (ChkErr(rc,__LINE__,u_FILE_u)) return
 461     ! adjust stopTime for mizuRoute clock
 462     stopTime = stopTime - timeStep
 463     call shr_cal_ymd2date(yy,mm,dd,stop_ymd)
 464     call shr_timeStr( stopTime, simEnd )
 465 
 466     call ESMF_TimeGet( refTime, yy=yy, mm=mm, dd=dd, s=ref_tod, rc=rc )
 467     call shr_cal_ymd2date(yy,mm,dd,ref_ymd)
 468     if (ChkErr(rc,__LINE__,u_FILE_u)) return
 469     call shr_timeStr( refTime, simRef )

A1: currTime and startTime use the beginning of time step, but stopTime use end of time step. simStart and simEnd (mizuRoute variable) use the beginning of time step (and for restart, simStart is restart time)

Q2: Does cime need to be downloaded??

A2: No need any more. Externals.cfg can be modified to exclude cime and mct and mpi-serial is not needed for stand-alone.

1. talked to Shervan about building model using existing libraries. Makefile change needed.
2. manage_external will be replaced with git submodule and be prepare for this. maybe no need for cime anymore, need only pio. no need for mpiserial.
3. work on finishing flooding, and managed reservoir turn on/off for lake.

Path to update mizuRoute in CTSM and CESM

1. Update ctsm add_mizuroute branch on izumi make sure works with the 5x5_amazon
2. Then you'll be able to use that when mesh files are replaced by hand for a case (on izumi and derecho)
3. Add the new mesh files into the ccs_config branch.
4. make a PR for ccs_config
5. make a PR for cmeps
6. Make a PR in cime to fix testing.
7. Those PR's come into CESM
8. I'll test a candidate version of mizuRoute on izumi and Derecho
9. make a mizuroute tag (Naoki needs to do)
10. the mizuroute tag goes into CESM tags
11. bring the mizuRoute branch into CTSM

During this process Naoki will be making other mizuRoute tags, and they will be brought into CESM when they are done as well as CTSM after we get to the last step.

**PIO writing bug **

Try PIO with regular netcdf (set pio_type to netcdf, currently hard-coded to pnetcdf in globaldata.f90)

when pio initialization, set some of these parameters

change PIO bufffer

Results in group case_last PIO_SPEC_FILE: /glade/work/erik/ctsm_worktrees/zendersource/ccs_config/machines/config_pio.xml

Results in group run_pio PIO_ASYNCIO_NTASKS: 0 PIO_ASYNCIO_ROOTPE: 1 PIO_ASYNCIO_STRIDE: 0 PIO_ASYNC_INTERFACE: ['CPL:FALSE', 'ATM:FALSE', 'LND:FALSE', 'ICE:FALSE', 'OCN:FALSE', 'ROF:FALSE', 'GLC:FALSE', 'WAV:FALSE', 'ESP:FALSE'] PIO_BLOCKSIZE: -1 PIO_BUFFER_SIZE_LIMIT: -1 PIO_DEBUG_LEVEL: 0 PIO_NETC PIO_BLOCKSIZE: -1 PIO_BUFFER_SIZE_LIMIT: -1

See CTSM #1206 for how lake evaporation precipitation is passed issue #157

• Replacing manage_external with git submodule soon. both cesm-coupling and standalone. Look at #450. no action needed

• MOSART refactor happening (Mariana is doing) and may want to look at new code structure. Look at. No immediate action needed

• Erik is working on CTSM branch add_mizuRoute (Erik folk) so that it works on derecho after tagged. Have to be aware of all (CTSM CMEP CDEP) updates so need to check the simulations (make sure no mapping error, etc.). Waite for Erik to work this out.

• Need to check with Brian Dobbins about Docker.

• two PRs #448(Erik add comments and then merge after the fix) #449(in progress)

• Mesh area. does it need to be in mesh or converted from sq-meter to sq-radian inside the code. Put area in sq-radian in mesh and used this instead of area (sq-meter) from river network data. Need to look into how the area in mesh is read in (now it is not read)

• I will keep mizuRoute input data (network and mesh) in my personal directory. and put them in cseg directory after tested to avoid many (incorrect) data there

• How do you use CESM-Lab to run an I compset with mizuRoute?
• How do you do docker on Derecho?
• can docker image work within some model workflow (standalone mizuRoute, maybe coupled with other models)?

• new tag: cesm-coupling.n02_v2.1.2

• Talk about Docker for Brian on Dec 6th (I gave him a testCast case).

• ctsm-mizuRoute (nag compiled) on izumi with the updated PR431 (remove pointers for io_desc_t data).

  Erik tested it and test passed, so merge it and tag it. Erik is running test on Cheyenne.

• is CTSM running on Derecho? mizuRoute compilation on Derecho too.

  For CTSM, see CTSM issue #1995. For mizuRoute, see issue #417.

  For stand-alone mizuRoute, need to update manage_externals. instruction : https://github.com/ESMCI/manage_externals/wiki#including-manage_externals-in-other-repositories. use manic-1.2.23 (instead of manic-v1.0.0 in the instruction)

• mizuRoute build instruction for novice (standalone).

• Docker (standalone)?

  look at CTSM docker for NEON site configuration in youtube. Brian Dobbins did for this. https://www.youtube.com/watch?v=xl73eC0VnMU&t=120s

• Naoki will need to be able to setup ctsm and run on izumi

cesm-coupling branch default?? think about it.

Will make another tag on mizuRoute. Erik will perform a suite of tests before mergin current PR #431

Naoki will also run a few tests.

Current cesm-coupling branch (as of Oct 25th 2023) should be identical to cesm-coupling.n02_v2.1.1 if irrigation in CTSM is off.

Then, everything is fine. merge PR #431. We left note on pioSystem definition in issue #438. if the tests fails, fix it before tag

• Update on Externals.cfg (and cime) for pio version.

  Updated to cime6.0.156 and pio2_6_2 in mizuRoute/Externals.cfg (no need to update mizuRoute/cime/Externals.cfg)

• How to deal with pioSystem (if using cesm-coupling, it is pointer, but if using standalone, it is variable)

  Try making pioSystem as pointer and pioSystem input argument used in subroutine (mostly pio_utils) needs to be pointer too.

• Every time PIO writes restart files, memory seems to increase.

  run ctsm-mizuroute with various restart and history write option so med.log file writes memory usage. and then maybe talk to Jim

• Derecho setup

  See for some information for ctsm: https://github.com/ESCOMP/CTSM/issues/1995 mizuRoute can be compiled with intel and cray-mpich. used thes modules.

  1) ncarenv/23.06 (S)   3) intel/2023.0.0        5) cray-mpich/8.1.25   11) proj/8.2.1  15) netcdf/4.9.2  17) cmake/3.26.3
  2) craype/2.7.20       4) ncarcompilers/1.0.0   8) udunits/2.2.28  10) fftw/3.3.10  14) hdf5/1.12.2   16) parallel-netcdf/1.12.3
  

  For debug mode (MODE=debug), need to turned off MODEARGS := #--enable-debug in lib/Makefile. cmake does not like --enable-debug. This will compile pio without debug mode, while mizuRoute is compiled with debug mode

• website using github page (any expert?)

  Did not talk about this

• Talk about new mapping problem we ran into.

• help from Bill/Mariana was critical, would still be working on it otherwise
• reasons for it: lack of error checking (including error checking that was in MCT driver), lfrac field in ROF was wrong (inland varied outside of ==1, up to 1.9 where 1.+small is max expected)
• things to change. This points to problems in our SE for CESM and even more issues with making CESM an even bigger effort. unit-testing, failure-testing, and breaking up code to make it understandable are going to be more important. We came from an environment where we expected people to be familiar with the entire body of code that they worked on (Naoki for example can do anything with mizuRoute and quickly find problems inside of mizuRoute, but when coupled to the complex CESM system with code he's not familiar with (and which I'm more familiar, but not expert in) we struggle.
• https://docs.google.com/presentation/d/1PCrTxui7ENsnHoYdBN7DiiY1gZziL1PchMAssGER8Fw

• Naoki re-run ctsm-mizuroute with HDMA network and HDMA-lake, w/ irrigation (unlimited and limited by rive volume) and w/o irrigation. Look at the results with Sean (feedback looks reasonable)
• Ocean coupling, had meeting have list of things to work on (See Aug 2nd list below), Sean is getting us one thing
• Lake coupling, Erik hasn't gotten to it, the mapping problem needed to be solved first
• Things to do: bring changes into CMEPS, update tag in CESM, update to latest for CTSM to bring it in there
• For Erik's time, plan to fix current problem, and then delay work until Decemberish time-frame. Need to work on SLIM in October (and finish it), Dust in November, mizuRoute next month...

Erik was able to produce correct mapped lfrac and remapped runoff in coupler history file with hist test case (including modified med_fraction_mod.F90).

Talked about the remapping issue. Erik/Naoki tested with modified CMEPS code (maptype = mapconsd -> maptype = mapconsf in med_fraction_mod.F90, only for ROF), and it appears that lfrac as well as runoff is mapped to ROF hru correctly.

Naoki will re-run with this modified code and update all the CTSM-mizuRoute runs, and these are a set of the cases

Enable mizuRoute to run at a specified time step independent of coupling frequency (PR #429). so far mizuRoute runs at the same as coupling frequency.

From last week

Naoki: Make two runs with hcru_hcru_mt13 grid and I2000Clm50SpMizGs compset. One is use online mapping, and another one is to use mapping file input (need to convert stand-alone mapping netCDF to SCRIP format).

I put the new slide on issue, and I will make runoff difference maps

Erik: is modifying CMEP and outputting land fraction (med_frac_rof_lfrac). Naoki needs to visualize the results.

Erik needs more time to look into this

Bigger question

• is lfrac for runoff normalization needed??
• Creating new network with simplified catchment so we can do ESMF mapping.

Change cpl code to incorporate different simulation time step. now simulation step is set to coupling time step.

Naoki: Make two runs with hcru_hcru_mt13 grid and I2000Clm50SpMizGs compset. One is use online mapping, and another one is to use mapping file input (need to convert stand-alone mapping netCDF to SCRIP format).

Erik: is modifying CMEP and outputting land fraction (med_frac_rof_lfrac). Naoki needs to visualize the results.

Talked about TSS website pointing to mizuRoute (need one sentence that link to mizuRoute website, which may need to be created)

Sean is looking into Greenland and Antarctica river network based on DEM, but just can test ocean-river coupling without these landmasses.

For the runoff regridding issue, try without normalizing remapped runoff by land fraction to see if imported runoff spatial pattern is good.

Computing ROF catchment area in radian^2 (divide area in m^2 by earth radius - 6.37122e6 m)

A few issues identified:

Runoff mapping issue again! Overall mediator maps runoff correctly but if you zoom in some of areas, some unexpected remapped-runoff spatial pattern (especially for hrus completely covered by CLM box). This is seen more in HDMAlk but also seen in HDMA to less degree. This is not severe as the wrong mapping seen in Jan 2022. reopen issue #xxx. Maybe turn on coupler history file to output import/export variables between lnd and rof.

Potential memory leak. Every time history files and restart files are created, the memory increase, even after cleaning pio decomposition and close file. Will create a pio toy model to demonstrate (use the latest PIO and version used in mizuRoute)

connected lake to the other river routing methods, but will give warning in a case script so this is still experimental.

Check how irrigation is on in CTSM.

We talked about the two grids that mizuRoute has and the challenges that they both will likely give. We need to make sure the small square one has the correct area's on it, and when we use nearest neighbor regridding that it works correctly. Also the complex grid might not be able to be used in Brian Kauffman's smoothing tool. As Mike explained the smoothing tool does a nearest neighbor remapping to the ocean, and then there is a ocean to ocean smoothing that happens. We also wonder if this can happen online by having CMEPS do a nearest neighbor mapping and then apply the smoothing map.

We thought that islands that mizuRoute has won't be a problem. But, we are concerned about Greenland and Antarctica because these are sensitive areas for the ocean model.

So these are the action items I saw:

• Erik: Get a SCRIP grid to Mike of the simple grid
• Erik: Test using ESMF nearest neighbor on the complex grid to see if that will work.
• Mike: Let us know if Brian Kauffman's tool is likely to work for this case.
• Mike: See if Brian Kauffman's tool will work with the simple grid
• Mike: Create an issue in CMEPS about doing the smoothing online
• Erik: Get a PR to CMEPS with the grid alias and the smoothed river to ocean mapping file
• Erik: Get a PR to CESM with a compset for this configuration with mizuRoute and MOM (we can still test without this by using a long compset name).
• Naoki: Look into merging the MOSART grid into the mizuRoute HDMA-lake grid over Greenland and Antarctica (Sean will help with this)
• Sean: Look into creating a mapping over Greenland and Antarctica based on a DEM that he has.
• Naoki: Look into other options of handling Greenland and Antarctica.

Some of these things may not work out, so we might have to meet with ESMF people or rethink the plan. So we'll meet again when we get to the next step or get stuck.

We went through some diagnostic plots in Jupiter notebook to evaluate several ROF cases (rHDMA, rHDMAlk, mosart)

• run with irrigation on
• evap, and precip coupling
• setup meeting with ocean group to explain what mizuroute grid looks like, and evaluate what needs to be done for coupling with MOM, and how hard it is
• improve direct-to-oulet speed (currently too slow)
• Greenland Antarctica, maybe use direct-to-ocean (so no need to do anything?)
• Look at cost estimate (high res, low res network, and with different simulation time step)
• Decide default option (rHDMAlk)
• notebook github (or improve usability)

Finished running those 4 cases from 1960-1999 where D19 data has the most valid flow data across the globe.

Now working on plots.

will merge #391 and #408 after Erik do final test. and tag mizuRoute

• Ran mizuRoute and mosart for 20 years. see /glade/u/home/mizukami/proj/ctsm-mizuRoute/run_case

  f09_f09_mg17_mosart
  f09_f09_rHDMA
  f09_f09_rHDMAlk_mg17
  f09_f09_rMERIT_mg17

  I will evaluate the results

• Go through Erik pull request #391 (to be ready to be reviewed soon)

• Go through abstract routing methods #405 (will be ready to be merged)

To see testlist ->

1. mizuRoute/cime_config/***.xml

2. use a script

cd CTSM/cime/scripts. 

##(--xml-compiler intel --xml-machine cheyenne)
./query_testlists --xml-category mizu 

./query_testlists --xml-category mizu --count -> gives you a number of tests

To run testlist ->

cd CTSM

#(no comparison with base simulation)
./run_sys_tests -s mizu --skip-compare --skip-generate

if you compare with baselines, need to provide location, /glade/p/cgd/tss/ctsm_baselines/ctsm5.1.dev114.mizu, can compare with baselines.

./run_sys_tests -s mizu --c ctsm5.1.dev114.mizu --skip-generate (no generation of baselines.  need a permission to store baselines, but compare tag, ctsm5.1.dev114.mizu)

• abstract interface.

• time management - sub-daily incrementing time variable (especially unit of day) gets slight off from the exact time stamp because of fraction of day. PR #398 fixes this.

• Create case with latest codes. regional case work??

  Right now ./create_new_case works with certain ROF grids. Grids not working are MERIT, cru_cru (will have to point the path to the correct grid)

Think about configuration file (now non-standard control file, but how the people like this?)

• current test status.

Test summary
35 Total tests
31 Tests passed
0 Tests compare different to baseline
31 Tests are new where there is no baseline
0 Tests pending
4 Tests failed

ERI.nldas2_nldas2_rHDMA_mnldas2.I2000Clm50SpMizGs.cheyenne_intel.mizuroute-default	  COMPARE_base_hybrid COMPARE_base_rest 		EXPECTED
ERP_D_P1x25.5x5_amazon_r05.I2000Clm50SpMizGs.cheyenne_intel.mizuroute-default	  RUN		EXPECTED
ERS_D_Mmpi-serial.5x5_amazon_r05.I2000Clm50SpMizGs.cheyenne_intel.mizuroute-default	  COMPARE_base_rest 		EXPECTED
SMS_D.5x5_amazon_rHDMA.I2000Clm50SpMizGs.cheyenne_intel.mizuroute-5x5_amazon_rHDMA	  RUN		EXPECTED

• pull-request Including t_bound variable in a history file and resulting changes #394

• time management - sub-daily incrementing time variable (especially unit of day) gets slight off from the exact time stamp because of fraction of day

• Talked about the status of testing Erik is doing. almost done.

• To start with ocean coupling, first trying creating mapping file between ocean mesh and rof centroid mesh with ESMF nearest neighbor

• Talked about creating abstract network routing (abstract class) and extend this abstract class to incorporate specific routing methods. This would reduce number of lines and simplify the codes (irf_route.f90, kwt_route.f90, kwe_route.f90, mc_route.f90, dfw_route.f90)

• MizuRoute doesn't handle ice-flow. Is it acceptable to pass ice direct to ocean in place?
  MOSART does not do anything = no route and all the zero in MOSART variable. Still have to move ice directly to ocean. Use direct-in-place and pass it to coupler. The coupler know which ocean grid based on nearest neighbor. maybe make sense to use direct-to-ocean for ice?

Maybe good to talk to Kate(?) about ice runoff from glacier (maybe only Greenland ice sheet). see https://docs.google.com/document/d/1W_Oq8A7T1kZ5eZtK0wQmrYJVAqWslVxE5BZ2CGjkLZ4/edit

• Are we handling all the negative flow as well as MOSART? The same as ice. QGWRL and irrigation flux can be negative and if negative flow is encountered, it use direct-in-place (to send it to ocean) or direct-to-outlet. For irrigation, there is a check to limit irrigation demand (i tried it before in mizuroute)

• Do we need to assess the global river budget for mizuRoute? Including if water is conserved in remapping? Is this done for MOSART or RTM?

Ocean model usually run for long time and see if water is losing or increasing (Keith is usually doing). Can get coupling variable to compared to RO variable.

• Who is the ocean contact we should work with to make sure mizuRoute works coupled to the ocean model? Mike Levy? Gustavo for mapping. Theoretically (technically) unstructured->ocean grid is expected to work. The biggest work is to create mapping between mizuroute grid and ocean grid. Use nearest neighbor mapping? see doc -> https://escomp.github.io/cism-docs/cism-in-cesm/versions/master/html/new-grids.html#generating-glc-ocn-mapping-files and issue https://github.com/ESCOMP/CMEPS/issues/334.

ROF->OCEAN. there is no overlap, so get nearest ocean grid box for each ROF grid.

• What needs to be done to MizuRoute to enable coupling to Hillslope CTSM?
  No need to do.

Talked to Sean before the meeting:

• Connecting CTSM-hillslope to mizuRoute sounds easy. Delayed runoff via hillslope process is stored in qover and qdrai gets zero and both are passed to coupler. Turn off mizuRoute overland routining when hillslope is on

• flood can be done, and feedback to land model (now flood is zero)

1. create new river parameter-bankfull depth (look at Yamazaki MERIT hydro data for this, and there may be river width data as well)
2. iterate reach routing. First round gets river height, and if this exceed, compute flood volume, and then adjust river outflow based on

Items:

• let's talk about qgwl split into qgw and ql (lake precip and evapo) to see if this is good way to handle lake water balance.

• Mapping issue is resolved (caused by coupler (CMEPS) changes)

• Every time mizuRoute is updated, need to run test list (we have setup and revisit this)

• tag mizuRoute and getting mizuRoute branch in CTSM into main CTSM branch.

• global domain test.

Pull request #363

Complete history restart capability (has not pull-requested yet). Need clean-ups (maybe separate pull request)

• using select type to compact the subroutines with different types for module procedures.
• need only one pioIOsystem for restart and history writing and reading? -> need only one initialization for PIO reading, writing history and restart
• type conversion from double to single before writing history variable. -> instead of storing sp in code, everything stored in dp and when writing with PIO, tell which precision is used.

Regional ctsm-mizuroute setup (klamath, S. Asia)

Items to talk about:

• pull-request #361 (separate answer change part from non answer change part)

• let's talk about qgwl split into qgw and ql (lake precip and evapo) to see if this is good way to handle lake water balance (with Dave)

• how to create case (not test case)? Yes, use ./create_newcase --case CASENAME --compset COMPSET --res GRID

• To test longer run w/ and w/o lake and compare with MOSART for global discharge (at discharge sites)

• short-term funding (due on 14th)

• question on CLM - PFT and CFT sum. NatVeg has to sum to 100% and CFT has to sum to 100%. landunit = PFT+CFT+glacier+wetland+urban+lake 100%

Merged pull-request #360 and #345

Erik updated CMEPS so it now reads the mapping files (it is temporary). I tested with updated CMEPS over PNW cases, and confirmed that runoff is mapped properly now. Erik mentioned new mizuRoute tag is needed to incorporate this into more permanent CTSM/CMEPS changes.

Naoki tested global HDMA case and confirmed mapping is not working.

Erik and I reviewed pull-request #353 - flexible output frequency capability. Erik approved it and I will merge after addressing the few minor comments (to-do)

Going through the cmep code to figure out where lnd2rof re-gridding occurs.

In component/cmep, created new branch called "rofregridtesting" and added ESMF_FieldWriteVTK in med_phases_prep_rof_mod.F90 before the med_map_field_packed call (re-gridding is happening there), but got compiling error so need more guidance from Bob.

Go through Erik reviews on pull-request #336 (make simulation time step independent of runoff time step).

Continue discussing how to debug mapping error with Bob. so no real progress on this. See issue #339

Worked on regional setup. Here - Pacific Northwest (PNW) case. subsetting river network data, land model surface and domain/mesh data. Creating mizuRoute centroid mesh, weight file

To create case for regional setup,

./create_newcase --case HDMA-lake_pnw --compset I2000Clm50SpMizGs --res CLM_USRDAT --user-mods-dirs /glade/work/mizukami/mizuRoute_data/ctsm_test/hdma-lake_pnw/user_mods --run-unsupported

store regional mizuRoute data (network, mesh and mapping file) under /glade/work/mizukami/mizuRoute_data/ctsm_test/hdma-lake_pnw/

This regional setup (PNW) is for debugging mapping issue seen in rHDMAlk, but useful for basin hydrologic study.

HDMAlk mapping issue not resolved yet. Need to talk to someone to get advice...

create subset domain HDMA-lake grid to test.

grid name = _ e.g., nldas2_nldas2_rHDMA_mnldas2, f09_f09_rHDMA_mg17

Joint with Naoki M., Dave L., Erik K.

• alias names including lake
• averaging
• LMWG talk
• Lake work
• Budget check
• Diagnostic tools?

Merged Shervan's pull request. water abstraction from evaporation may cause water balance error when lake water volume is smaller than evaporation.

• LMWG workshop short abstract checked

• time-average output.

  Almost finished coding and some testing. There are bunch of rules for time averaging associated with simulation step, history file frequency. Restart reading has not been implemented.

  Q: Does restart information on aggregated field needs to be written always or just continuous run?

  A: CTSM/CESM uses rh files that only include aggregated fields and number of time steps, separated from r file. They are only output when needed. If h0 is monthly, then rh0 would be only be output if it wasn't on a month boundary.

• test data for stand-alone mizuRoute.

  check out https://git-lfs.com/ For Cheyenne, need to load module

  CTSM example: python/ctsm/test/testinputs has some test files that we use lfs to manage.

  or explore other options (putting mizuRoute git repository etc.)

• branch protection. is it good to set and what restrictions? https://github.com/ESCOMP/mizuRoute/settings/branches.

• Fortran questions: MOSART uses pointer in derived data type member. dose this help reduce memory?

• PIO_Sync: what is benefit of using this??

• output time-aggregated variables.

  • Current: specified in control file -> NewFileFrequency (daily, monthly, yearly and single) tag and direct T/F for each variable.

  • Using namelist (similar to CTSM) good idea? how to maintain backward compatibility.

Decided to output one history file with user specified time average (average only for flux and volume, could be instantaneous for volume)

1. Issue #330 was not closed by merging PR #329 (this is because trying to merge into non main branch. so need to close manually).

Now, ctsm-mizuroute with lake works so Naoki will test and look at results.

1. To get ready for the latest setup, I will have to fetch and merge CTSM update branch (add_mizuroute branch name from Erik repo) including ccs_config and run manage external, and also need to update mizuRoute cesm-coupling. the ccs_config branch is mizuroutelakegrids

2. I will inform Will that I am interested in updating ctsm mizuroute with lake in winter workshop or maybe CLM meeting (if there is free time) mid-late January. Need to sign up sheet for presenter (will be distributed in the end of December 2022)

3. Erik continues working on passing Precp and Evap and once these are passed and used for lake water balance, it is ready for merging into CTSM main branch.

Alias name finalized: use full acronym _rHDMA, _rMERIT, _rHDMAlk, _rUSGS.

If different version or resolution (defined by minimum hru area), think about later.

Vector river network resolution may use minimum catchment area.

Erik is updating CTSM branch (it will have new alias name including _rHDMAlk, and new config name convention, e.g., f09_f09_rHDMA_mg17 instead of f09_f09_mg17_rHDMA). and let me know.

Debugging Erik's HDMA-lake runs:

1. update on river-lake network (hru id order needs to be matched with mesh), and point to correct one (was wrong network)
2. control file change - need to add is_lake_sim tag to switch on/off the lake.

Looks like mizuRoute with HDMA-lake network is running. Next step is to update PR #329 (include is_lake_sim tag in control file) and run with lake on.

Talked about grid alias and case name for mizuRoute HDMA with lake configurations for cesm We suggest rHDMAlk as alias name, and HDMA_lake_mz for case name.

After incorporate these names in to cesm, update ctsm (fetch and merge) and run manage external. ./manage_externals/checkout_externals ccs_config

Erik give a final instruction

Started creating river lake dataset.

New grid alias names.

1. hcru_hcru_mt13_rHDMALake
2. f09_f09_mg17_rHDMA
3. 19_f19_mg17_rHDMALake

possible component name: HDMA-LAKEmz

• ESMF mesh was created: /glade/work/mizukami/ocgis_conversion/output/ESMF_unstruct_HDMA_hydrolake_global_square.nc

• Three ESMF mapping files need to be created - 1) hcru vs HDMA-LAKEmz, 2) fv1.9x2.5 vs HDMA-LAKEmz and 3) fv0.9x1.25 vs HDMA-LAKEmz

• Network: Network_topology_HDMA_HydroLake_v1.nc

If you call lake alias when creating case, will have to adjust control file, so lake mode is on in mizuRoute

Global water balance error check (PR #326). Need to test with CTSM

To-Do: create HDMA-HydroLake catchment mesh for CESM. evaporation and precipitation import through NUOPC

Met with Dave. Three items to focus on for the rest of year.

1. Budget check: Look at RTM also and compute and print out global error [m3] and abort if it is greater than 10-6. Some work in mizuRoute (PR #319) is done and finish this up.

2. Lake implementation: focus land-only coupling. This is the highest priority. The target date for finishing this is end of Nov. plan to show some results in CTSM science meeting (some time between the end of Nov and beginning of Jan) and winter CESM workshop. The results == comparison of discharges (big river) between 1) mizuroute and MOSART and 2) mizuRoute w/ lake and w/o lake, computing cost, and so on.

3. Temporal aggregation: higher priority to improve usability especially for longer run.

4. Diagnostic tool is useful.

Met with Sean and went through this ppt (0.5 degree routing with MOSART and mizuRoute). ctsm_irrigation.pptx

Sean is setting up HUC12 CTSM over western US (Naoki share NLDAS domain HDMA catchments with Sean to see if he can simplify the HDMA catchments so that online mapping is possible in CTSM).

Naoki is working on water budget analysis similar to MOSART.

• Setup 0.5 degree gridded network for mizuRoute with various CLM grids. No need to create alias When you create a case, specify only clm grid like f09_f09_mg17 or case name like ./SMS_Ly3.f09_f09_mg17.I2000Clm50SpMizGs.cheyenne_intel.mizuroute-default

• Update CTSM branch mizuRoute is now pointing to.

• New pull request #315 to finish up direct_to_outlet. Erik reviewed, so merge after address the comments.

• How to create mosart case use compset I2000Clm50Sp e.g., SMS_Ly3.f09_f09_mg17.I2000Clm50Sp.cheyenne_intel.mosart-default

• Erik started working precipitation and evaporation import into mizuRoute from CLM in cmeps

• How to evaluate water budget... See Sean after 0.5 gridded runs finish (next week)

To-do lists

• Setup 0.5 degree gridded network for mizuRoute with various CLM grids. Create alias

• Update CTSM branch mizuRoute is now pointing to.

• Merge pull request #313

• (a month to few months goal) Pass precipitation and evaporation for lake, and also start looking at new river-lake data.

initial code - direct_in_place, direct_to_outlet

Need to evaluate the simulation (what should I look at)

direct_to_outlet is slow. need to optimize the algorithm

ctsm_irrigation.pptx

After meeting with Dave, we decided to do the followings:

• Run mizuRoute and MOSART with 0.5 degree gridded network (both MOSART and mizuRoute gridded network has the same flow direction, and land mask), and compare the simulations w/ and w/o irrigation and w/ and w/o irrigation threshold, and negative flow handling option.
• Run the results with Sean. if qualitatively similar, move on to lake implementation.
• Then maybe come up with better evaluation strategy for irrigation, and negative flow handling
• Implement water budgets in mizuRoute.
• Implement direct_all_outlet (bypass routing and send all the flow to the outlet). Sean's suggestion a few month ago

Initial version of direct_to_outlet option is completed. mizuRoute domain decomposition can assign tributaries and their mainstem where outlets of the tributaries are located to different tasks, so MPI communication is used to send negative runoff to the outlet (from one task to another).

Tested with two LND-ROF grids: f19_f19_mg17_rHDMA and f19_f19_mg17_rMERIT.

Need to check the simulations, then will need to speed up this process (checked the timing of direct_to_outlet option. over 90% of total rouitng time is spent on sending fluxes to outlets).

Also, need to turn on clm option: limit_irrigation_if_rof_enabled = .true. and change irrig_threshold_fraction = ?? (check CLM namelist documentation)

• Difference in CLM runoff (QDRAI+ QOVER+ QDRAI_PERCH) between runs with irrigation-off and irrigation-on (see image below- 2002-06 mean runoff, run w/ irrig minus run w/o irrig). It seems to be odd that CLM runoff with irrigation on increases runoff over CLM runoff with irrigation off. Need to turn on threshold in CLM (in last meeting with Dave and Sean, I misunderstood "threshold" (I thought threshold should be on in ROF option not CLM option)??

• Concern about mismatch between mizuRoute HRU polygon and CLM grid along the coast line, especially the case where a CLM grid box is not overlapped by any mizuRoute HRU polygons at all. What will happen to the CLM runoff at those grid boxes?? Runoff is sent to the ocean directly (without passing mizuRoute)? The case where the mizuRoute polygon is not overlapped by any CLM grid boxes and the case where mizuRoute polygon is covered partially by CLM grid boxes are probably ok. But need to check. See Issues #301

• Need Greenland and Antarctica river network for fully-coupled run. Maybe patch the MOSART gridded network to HDMA and MERIT-hydro. MERIT-hydro has Greenland river network, not Antarctica.

• What will happen to discharge (routed runoff by mizuRoute) that pours to the Endorheic basin (e.g., great basin). For now, mizuRoute routes the runoff to the outlet regardless of whether it's outlet to the ocean or Endorheic lake. Not sure if runoff at Endorheic outlet needs to be sent to the ocean somehow?

• codes for direct_in_place for negative runoff (qgwl and irrigation) are merged at PR #297.

• now working on direct_to_oulet option.

Discussion item for next meeting:

• Talk with Dave about irrigation working now

• Run SMS_Lm1.f19_f19_mg17_rHDMA.I2000Clm50SpMizGs with irrigation and no irrigation for a month (1980-01)
• ctsm_irrigation.pptx
• ctsm_irrigation.pdf

Apply limit??

• Talk with Dave about VOLR for headwaters top or bottom of HRU

• mizuRoute puts runoff from CLM at an outlet of HRU (see overall mizuRoute procedure). Because of this, reach routing is not performed at headwater HRUs, and no volume is computed. This needs to be resolved.

Sean said, zero volume for headwater would be probably ok, and try testing.

• Talk with Dave about QGWL and how it should be handled. See #164

• Now Flrl_rofgwl is imported into mizuRoute, but not doing anything with it.

handling qgwl is essentially the same as irrigation. need to send negative q to ocean.

• Talk with Dave about priorities:

1. QGRWL
2. Lake work (need to pass Precip and ET through coupler, Erik will help with this)
3. Lake network data (need to talk with Shervan)
4. Send back amount of water removed from Lakes? Or is VOLR sufficient see -- #170 (this is longer term issue correct?)

Finish up QGRWL and irrigation are higher priority and make sure that water balance is close. Implement all direct to ocean option to test all runoff from CLM == water going to ocean. For lake, will need to work out details....

• Will asked me if I need LMWG CSL allocation for 2022-2024. if so need some short paragraph for the request by Friday.

This is development work, so no worry for this (LWMG allocation would be run more for science)

No meeting due to other priority (Erik) and traveling (Naoki)

Talked about how to turn on irrigation in CLM. Make sure if these are turned on, mizuRoute works.

CLM namelist

• irrigate = T

Other related namelist variables:

• irrig_river_volume_threshold

• limit_irrigation_if_rof_enabled

• use_groundwater_irrigation

Irrigation:

Looked over irrigation part in MOSART, a few questions came up:

1. After irrigation demand is imported to ROF space from CLM, where does mosart take out the water from? Does it take water at the same location as irrigation demand (crop area)? or realistically taking it out elsewhere (e.g., somewhere upstream large water body or nearest river mainstem)?

2. If irrigation demands are too much (more than water volume in river), actual water taken out is communicated back to CLM? (MOSART does not seem to do that)

Other minor discussion:

In fortran, is it ok to supply a portion of array (i.e., array(lower:upper)) into subroutine or function argument? does compiler make and copy temporary array in memory, which likely slows down.

Currently mizuRoute outputs discharge (and other variables) at all the river reaches in the network. I have a few global river discharge at gauges.

• Dai, A. (2017). Dai and Trenberth global river flow and continental discharge dataset. https://doi.org/10.5065/D6V69H1T

• Beck, H. E., de Roo, A., & van Dijk, A. I. J. M. (2015). Global maps of streamflow characteristics based on observations from several thousand catchments. Journal of Hydrometeorology, 16(4), 1478–1501.

D17 data has ~900 sites (large river outlet) B15 data has ~21,000 sites (including inlands)

Does it make sense to output only at those sites? -> would be nice to have user output options - all reaches or selected reaches at gauges.

Wish list: Create diagnostic packages. look at examples and replicate this for mizuRoute ouptu, then build new hydrology specific. This is probably not high priority

higher priority is 1) irrigation, 2)enable lake, 3) tracer to route ice (snow cap in clm). then ocean coupling

Other important task is to send total water mass back to clm.

Reviewed mizuRoute overall procedure and hillslope hydrology (See Apr 20th notes) for brainstorming how to couple both. If the spatial resolutions of mizuRoute and hillslope hydrology are the same, that would be ideal. if not, remapping could still work technically.

Reviewed coding guidelines (new wiki page)

Naoki: Merge #274. Work on incorporating gauge meta.

Erik: test on Izumi.

With Sean

The hillslope hydrology option needs flood information from the river model. This may require mizuRoute to use a compound channel (or use river channel depth) and compute flood volume when routed flow depth exceed the depth. The river depth for each reach as a river parameter is required, or estimated with some empirical equation, just like river width computed as a function of total upstream area now.

Will have to handle negative flow and irrigation, which are imported from CLM. Currently mizuRoute is able to inject or extract water from any reaches. mizuRoute needs to know which reaches, how much and when (this can be from user - time series of water injection/abstraction or CLM through the coupler).

It would be good to create test case for a basin (e.g.,Columbia, Colorado River) for Sean to to be able to test CTSM-mizuRoute

From Sean. river-hillslope interaction concept

With Dave L.

Naoki -- still trouble with the build? I did get the openmpi intel build fix PR to go into cime.

Naoki -- Tried the latest CTSM (cloned on Apr 14th) master branch, with SMS_D.f09_f09_mg17.I2000Clm50SpRs.cheyenne_intel.clm-default, and failed.

NLDAS-USGS case may require less CPUs. I tested with SMS_P360x1_Lm1.nldas2_rUSGS_mnldas2.I2000Clm50SpMizGs.cheyenne_intel.mizuroute-default and was able to run

It would be good to talk to Sean about hill-slope and river interaction. River model needs to track, discharge, volume and water depth. water depth may be needed for hill-slope and river interaction.

NLDA-USGS issue. need to describe what happen in issue (Erik) and Naoki add the cause (need to look at the code)

Warning: external config change. mizuRoute points to the specific tag and the lates branch is commented out now in .gitmodule

gitHub auto close issue when merging the pull-request. Need to add close (fix, or resolve) #issue number in main comment section. Then if the pull-request is merged, the issue tagged will close.

Funding for 25% of Naoki's time for CESM work will start after May 15th. We will normally always do these monthly meetings, but will often be short for the update to Dave. The agenda will be a normal kind of standup meeting to give an update of what we each are working on, as well as anything we are stuck on, that we need to get help with.

Erik should do things needed for Naoki to make progress in mid May. One issue we identified was for the lake fields to be passed in the coupler.

• Dave tells me mizuRoute's importance is that we don't have in-house ROF component expertise. so hopefully mizuRoute can handle that. mizuRoute also allows us to run on HRU's which is likely important for some types of work for CTSM.

• Ok, good to know.

• cime issues and PR.

• Included off-line mapping file in grid_config. can merge soon.

• mizuRoute issues for coupling

• mpi-serial issue

• Running with different number of threads for different components? Does that work for nuopc now?

• yes. should be able to do. need to try.

Items to talk about

1. hybrid run CTSM
2. Update on incorporating recent CTSM update

Identified next tasks

• 1. create hcru_hcru_HDMA and hcru_hcru_rMERIT grids.
• 2. Incorporate another CTSM update.
• 3. Remove time dimension from restart file, but write timing information.
• 4. Check if MPI-serial works to compile without MPI library.
• 5. Slowly work on tracer (maybe).

Meeting with Jim and Mariana

PIO history file writing was resolved at #203

pioSystem was initialized at CIME, and do not need to be initialized within mizuRoute. mizuRoute was confused between pioSystem initialized at CIME and the one initialized within mizuRoute

1. PIO issue due to cime update in ctsm. cannot close netcdf.

2. Area correction. issue #183. very low priority. this would be needed for inconsistent projection of the grids across the model components

1. Erik continues on CTSM updates and close to finishing testing.
2. Naoki makes mapping file between 0.5 LND grid and river network.

• 1. ctsm update.

Erik will work on this and let me know. Once it is done, 1. Pull ctsm from Erik. 2. run manage_external/checkout_externals. 3. pull cime.

• 1. short archive issue.
• 2. restart test with ERS issue.

Erik fixed. ROF name needed to be mizuroute instead of mizuRoute. merged into pull-request NCAR#181

Fixing short-archive issue fixed restart test. restart test passed

• 1. restart test with ERS issue. update from Erik.

Looks like cime issues and need further investigations

• 1. restart test with ERS did not go successfully (though looks like it did restart)
• 2. how to set new history frequency.

Just to be informative and compatible with other components. Include current time step to start in restart file, and read in and add one. Problem in restart test is due to the ROF short-archive problem Erik will fix and push it to mizuRoute cesm-coupling branch (in my personal). This would fix restart test.

It is more complex than I thought- history file vs restart file timing. If restart happen in the middle of month and history file is monthly, restart run writes in the existing file not new file.

Use these namelist values to control how history file is written (common way across the ctsm components)

hist_nhtfrq: the frequency of writing hist_mfilt: how many of those time-samples go on the file before you finish writing.

e.g., if hist_mfilt = 1, only write one time sample per file. If hist_mfilt = 5 write 5 time samples on each file.

For example if frequency is daily. and hist_mfilt=365 then put daily values in a whole year worth on the file. if frequency is yearly, and hist_mfilt=1 then that would put the yearly average for each year on one file.

mid-term plan to do this.

• 1. New mapping between lnd (1-degree) and ROF. 1) fv0.9x1.25 (lnd), MERIT_Hydro (rof), 2)fv0.9x1.25 (lnd), HDMA (rof). /glade/work/mizukami/py_mapping/mapping_conversion/data/
• 2. How to activate restart writing?
• 3. Variables sent back to CLM from mizuRoute. rtmCTL%volr, rtmCTL%discharge, and rtmCTL%flood. How are these used in CLM?

Wait for Erik to correct names.

Use ERS instead of SMS when making test case: e.g.,

./create_test ERS_Vnuopc_Lm5.f19_g17_rMERIT.I2000Clm50SpMizGs.cheyenne_intel.mizuroute-default -r . --no-build --queue premium --walltime 01:00:00

Need to implement rpointer.rof reading statement in code and assign restart file name in mizuRoute.

Don't worry about now. They are not used in CLM.