Meetings notes 2023 Software - ESCOMP/CTSM GitHub Wiki

General outline of agendas:

  • Assign timekeeper

  • CTSM Software wins for the week

  • FATES Update

  • Questions and issues: quick things that people need help with

  • Short term issues/decisions

  • Longer term issues/decisions

  • Information gathering that's needed

  • Brainstorming on new ideas

  • Go through issues: next, high priority

  • Go through upcoming tags

Future events for the year:

  • Jan 5th, 2023 -- First meeting of the new year
  • Jan 12th, 2023 -- What needs to be done by the LMWG meeting? Winter Qtr
  • Jan 26th, 2023 -- Last meeting of January
  • Feb 9th, 2023 -- What do we need to adjust based on the LMWG meeting?
  • Feb 23rd, 2023 -- Last meeting of February
  • Mar 30th, 2023 -- Last meeting of March
  • Apr 13th, 2023 -- Spring Qtr
  • Apr 20th, 2023 -- SEA Conference cancelled
  • Apr 27th, 2023 -- what did we learn from the SEA conference? Last meeting of April
  • May 4th, 2023 -- Star Wars Day, More on SEA conference
  • May 18th, 2023 -- What do we need ready for the CESM workshop?
  • May 25th, 2023 -- Last meeting of May
  • Jun 15th, 2023 -- cancelled, CESM Workshop
  • Jun 22nd, 2023 -- What do we need to adjust based on the CESM workshop?
  • Jun 29th, 2023 -- Last meeting of June
  • Jul 13th, 2023 -- Summer Qtr
  • Jul 27th, 2023 -- Last meeting of July
  • Aug 31st, 2023 -- Last meeting of August
  • Sep 28th, 2023 -- Last meeting of September
  • Oct 5th, 2023 -- First meeting of FY2024
  • Oct 12th, 2023 -- Fall Qtr
  • Oct 26th, 2023 -- Last meeting of October
  • Nov 23rd, 2023 -- Cancelled Thanksgiving
  • Nov 20th, 2023 -- Last meeting of November
  • Dec 14th, 2023 -- AGU
  • Dec 21st, 2023 -- Last meeting of December
  • Dec 28th, 2023 -- Cancelled Christmas
  • Jan 4th, 2023 -- First meeting of the new year
  • Jan 11th, 2023 -- What needs to be done by the LMWG meeting?

Meetings after April 20th

See Google Doc: CTSM Software Engineering Meetings 2023

April 20th, 2023

  • Timekeeper: Bill

Agenda

CTSM Software wins for the week (5 minutes)

  • Ryan's PR with BGC call sequence

FATES Update (10 minutes)

  • Still working on #1959 (update to use BGC call sequence)
  • FATES refactoring - still B4B, working on unit testing. For later - how should these kinds of non pass/fail tests get called?
    • sub-question: FATES parameter file read-in subroutine is in the CLM codebase. This seems odd to me. Feels like FATES should have a parameter read-in subroutine that CLM calls. Ultimately, just makes unit-testing a bit more annoying the way it's set up now.

Meeting Housekeeping: How to improve these meetings? (10 minutes)

  • Sam R. -- sample Google agenda. Lead us through how this works

Questions and issues: quick things that people need help with (1 minute)

  • Keith: What's the best way to make a domain file for an atmospheric forcing dataset?

Short term issues/decisions (5 minutes)

Longer term issues/decisions: (5 minutes)

Upcoming tags/Next issues: (20 minutes)

Focused discussion (15 to 30 minutes...)

  • Ryan's BGC call sequence PR

Making a domain file for an atmospheric forcing dataset

Keith: a user on the forum has their own atmospheric forcing dataset; how can they create a domain file?

If you use mknoocn, it will set the land mask to 1 everywhere. Erik thinks that may be what's done for current atm forcing datasets, so that makes sense as long as the dataset defines data everywhere. So can use that process together with other pieces of the process when creating lnd domain files.

April 13th, 2023

  • Timekeeper: Adrianna

Agenda

CTSM Software wins for the week (5 minutes)

FATES Update (10 minutes)

  • Restart memory reduction updates (FATES#991), no answer changes
  • Still need reviewers for #1959?
  • FATES refactoring - all B4B so far, figured out how to add netcdf to CMake!

Meeting Housekeeping: How to improve these meetings? (5-10 minutes)

  • Use an app for timing? So we can see how well and what we follow?
  • Agenda in google doc -- so can co write? This wiki could point to a different doc for each meeting or to meeting notes for the entire year. The google doc could be open for editing from the group that normally attends and global read-only for anyone with the link.

Questions and issues: quick things that people need help with (1 minute)

Short term issues/decisions (5 minutes)

  • Next steps for SNICAR updates PR #1861

Longer term issues/decisions: (0 minutes)

Upcoming tags/Next issues: (20 minutes)

Focused discussion (15 to 30 minutes...)

  • FATES use the normal BGC call sequence #1959

Meeting housekeeping

Would it be good to use an app for timing (visible to everyone)?

  • Could be worth trying
  • Sam: it feels like the most important thing is to see how much time we should allot to each section – e.g., what tends to take longer / shorter than the allotted time so we can make better time allotments moving forward.
    • Sam's sense is that FATES updates could often use more time
    • Will's sense is that short-term issues often take longer than allotted

Short-term vs long-term:

  • Idea was that short-term things need input this week; long-term could be deferred until there's time in a meeting to cover it.

Sam suggests: Throughout the week, people can add items they want to discuss; indicate how long they think it will take and the priority; then it can be put in the appropriate section.

Using a google doc for meeting notes:

  • Sam suggests having a google drive folder with a separate document for each week's meeting.
  • Advantages would be that everyone can view and edit this at once
  • A disadvantage could be searchability
  • Pros and cons to having a separate document per week or one per year (separate documents for each week can make it easier to find the week when we discussed some particular thing based on a search for multiple words; one document per year can make it easier to quickly scroll / search through)

Review of Ryan's PR

https://github.com/ESCOMP/CTSM/pull/1959

Ryan feels like the main question points on this have already been addressed by email, etc.

High-level: There are things that CTSM does in terms of tracking C & N that, up until now, FATES has not participated in. Instead, there were conditionals leading to a subset of things being done with FATES. This PR changes it so that previous conditionals on use_cn now check use_cn or use_fates_bgc.

Movement of summary variables (like totc_col) into the soil biogeochemistry type: This was needed because the type that initially contained these variables isn't allocated when running with FATES. We had discussed the possibility of creating a type just to have these summary variables, but Ryan's sense at the time was that people were okay with having these in the soil type.

Regarding CNVegetationFacade: Previously, FATES wasn't using this. Now FATES is using at least parts of it.

  • Bill: the initial vision for this was that FATES might have its own version of this module, in an object-oriented sense. We had a bit more discussion of this as a longer-term thing; not clear if that would make sense or not.
  • Ryan points out that this is entangled with CNDriver calls... this would need to be disentangled if we wanted to go the object-oriented route.

Summary: Both C & N code is in place, though N won't do much yet. Harvesting code is also written to accommodate FATES harvesting fluxes and run harvesting code - specifically referring to the product pools; currently FATES is passing 0 fluxes, but now the hooks are in place to allow non-zero values.

April 6th, 2023

  • Timekeeper: Will

Agenda

CTSM Software wins for the week (5 minutes)

  • FATES tag and FUNITCTSM test working

FATES Update (10 minutes)

  • Split associate statement to avoid NAG compiler line continuation problem (FATES #1009)
  • Migrate FATES to use BGC call sequence #1959
  • The Great Refactoring

Questions and issues: quick things that people need help with (5 minute)

  • Sam: Should I attend the SEA Improving Scientific Software conference?
  • Keith: Do we need a project board for CESM3 Development simulations? Larger discussion, we'll decide on a subgroup to start, and then present to both CTSM software and CTSM science.

Short term issues/decisions (?? minutes)

Longer term issues/decisions: (3 minutes)

Upcoming tags/Next issues: (20 minutes)

Focused discussion (15 to 30 minutes...)

March 30th, 2023

  • Timekeeper: Sam R.

Agenda

CTSM Software wins for the week (5 minutes)

  • Bill's tag with a difficult externals update is in, Dev_120!
  • Dust work is close, will be meeting next week with CAM-Chem team.
  • mizuRoute is close to being able to come in.

FATES Update (10 minutes)

  • Who is going to do the FATES updates with Jackie out of the picture? Likely Adrianna, but Greg/Ryan can fill in.
    • Adrianna also planning to play (even more of) a leadership role CTSM-FATES development (both science and SE).
  • Little to report out, lots of travel and absences on the FATES team.
  • Soil moisture initialization changes with FATES are close to coming in for CTSM #1962.

Housekeeping (10 minutes)

  • Erik: To send out website re. GitHub ssh changes.
  • Erik: Talk about how to signal moving on for time-keeping in our meetings. How do we signal in a way that is comfortable to everyone? How do we make sure we aren't cutting people off? Or being inequitable in checking time?
    • Rotate responsibility around the group (similar to note taking), We'll add this to the agenda, rotating alphabetically.
    • Visual and auditory cue are helpful (although maybe too formal for this situation.
    • Adding a countdown clock to desktop so everyone sees it.
    • Raising hand could be a good way to do this?
  • Sam: Can we designate an agenda item to space for "Here's a question/issue I've run into; can someone help me with it later?" (I put mine in "Short term issues/decisions" but it doesn't feel right there.)
    • Add heading for Questions/issues to raise (similar to the standup's meeting).
  • Erik: Next week at 10 is an SEA general interest meeting, can we run a short meeting next week?
    • Yes.

Questions and issues: quick things that people need help with (1 minute)

  • Sam: How to set up a custom single-point run with release-clm5.0?
    • Keith posted this, which Sam said he tried, but ran into issues.

Short term issues/decisions (NA)

Longer term issues/decisions: (3 minutes)

  • Conda environment issues: #1974. Do we need to develop ways to prevent this, or force checks in ctsm_pylib?
    • Negin had some suggestions on this issue thread.
    • Keith and Erik are going to meet on this.

Upcoming tags/Next issues: (20 minutes)

  • FATES soil moisture will be coming in next,
  • Regional subsetting to follow (Erik said it's close)

Focused discussion (15 to 30 minutes...)

  • Quick updates on coupled model simulations with CPL_HIST, #1951
    • Resolving issues related to solar zenith angle.
    • Air density being written over by CTSM (not cpl_hist), Keith has a hack.
    • Keith has a number of code mods that need to be brought into main to ctsm, cmeps, etc. Keith will work with Bill regarding timing of when this comes in for spinup and running latest CESM tests.

March 23rd, 2023

Agenda

CTSM Software wins for the week (5 minutes)

  • Not, a win, but Jackie is moving on to another great opportunity for her. We will miss her though and her contributions.
  • Adrianna helped me get to the next step
  • Great candidates for the next CGD director.
  • We are 99% sure we figured out the mapping issue with mizuRoute

FATES Update (10 minutes)

  • initialization problems - nocomp with large trees (FATES PR#995, more water (PR#1962)

    • needs more discussion on CTSM side?

  • FATES to use BGC call sequence PR#1959

    • in testing, needs reviewers

Longer term issues/decisions: (9 minutes)

  • graph and nodes for CTSM refactoring (Adrianna & Erik)

Upcoming tags/Next issues: (20 minutes)

Ryan's FATES PR (1959)

Small diffs in TOTN and TOTC. Feeling is: as long as this stays near roundoff (verifiable with summarize_cprnc_diffs), this is fine, since this is expected due to a change in order of operations.

FATES change in initial soil water (1962)

Feeling is: for now, we can let this stay as is, with different initialization for FATES. We'll open a CTSM Discussion to decide what we want to do long-term.

  • Actually, let's use an issue to ensure that this doesn't get forgotten, and because we will likely want some code changes to emerge from this.

Greg will bring this home.

Graph and nodes for CTSM refactoring

Context: Gordon has been talking about bringing in the multi-layer canopy, but it requires some refactoring: currently, fluxes are calculated in different places, etc. In addition, Adrianna has been thinking about reworking things like albedo calculations to have them shared between CTSM and FATES.

This has led Adrianna to look into possible solutions for generating graphs showing relationships between subroutine calls as well as variables: how they're set, where they're used, etc. Is there some way to use the abstract syntax tree for this?

Some ideas:

Adrianna will look into this more.

Compiler warnings

Ryan wonders if there would be value in creating a count of warnings in each PR so we can keep track of whether this is going up or down.

Ideally we'd clean them all up, but that would be a big project.

Erik: used to use FLINT for this.

March 9th, 2023

Agenda

CTSM Software wins for the week (5 minutes)

-- Dave's back!

FATES Update (10 minutes)

Short term issues/decisions (1 minute)

  • Erik: ctsm5.1.dev119 -- Peter is reviewing the code to make sure it makes sense with the changes that have happened. I highly support doing this, but think we should do the tag as is now. If Peter finds issues we can still make those changes on the ctsm5.2 branch.
  • Erik: Sent a meeting invite for Friday to talk about github actions to Greg and Sam R.

Longer term issues/decisions: (9 minutes)

  • Erik: Bringing up issue for Bill that Naoki is using OCGIS for creating mesh files for mizuRoute. Since Ben is long gone this is a real problem for us long term.
  • Ryan/Will: Hardcode nitrif_denitrif flags to .true.? This is problematic for a smallville test. Was an issue for FATES, but that's been changed.

Upcoming tags/Next issues: (20 minutes)

  • xxx

Focused discussion (15 to 30 minutes...)

  • xxx

Soil moisture initialization with FATES

To get more reasonable results with FATES, it seems we need higher initial soil moisture. The current implementation involves having different initial values for FATES vs. non-FATES.

Some of us feel that we should try to avoid having arbitrary differences like this: it can throw things off for users who – e.g. – are doing comparison studies of FATES vs non-FATES. A solution could be to increase the initial soil moisture for non-FATES as well as FATES.

Sam: if it isn't possible to use the same initial soil moisture for non-FATES, one possibility would be to require an explicit flag to turn on this difference.

  • Erik: or have a namelist-settable initial soil moisture value.
  • Bill: feels it would be better to just change things consistently for non-FATES and FATES... then just consider one of these namelist options if that causes problems.
    • This will probably require a run to ensure that this doesn't break things.

Next steps: Discuss this in a more science-focused meeting so we have the right people in the room.

ctsm5.1.dev119 - gross unrepresented land use

Plan is to bring this into master with some small test cases. Then with the 5.2 integration it will be more heavily exercised.

Will plan to bring this in as is, then make any changes needed on the 5.2 branch or directly on master.

Agreement that this makes sense. Probably wait until Monday-ish and then move ahead.

OCGIS

Erik: Bringing up issue for Bill that Naoki is using OCGIS for creating mesh files for mizuRoute. Since Ben is long gone this is a real problem for us long term.

There is currently a problem with how multi-polygons is done; talking with Bob about this. It seems like OCGIS may have problems with how it creates multi-polygons from the shape files.

mizuRoute starts with a shape file and converts it to a mesh; OCGIS does this.

Bill: the unfortunate reality is that the ESMF team has lost its python developers, so it's hard to promise any significant support for this package. There is some hope that this could change long-term – once we get an ESMF core team manager – but for now things are uncertain.

Sam: can the GDAL tools handle this? Erik will point this out to Naoki.

Taking out nitrif_denitrif

This is no longer an issue for FATES.

This is still set to false for a smallville test... but it looks like it may have been removed... or if it's still there, it might have only been done so we'd have a test of this configuration... so can just change / remove the test.

March 2nd, 2023

Agenda

CTSM Software wins for the week (5 minutes)

-- Our testing caught a change in answers in a diagnostic field in the GULU branch. By the way the GULU branch is a PR that was outstanding for 5 years.

FATES Update (10 minutes)

Discussion items: (15 minutes)

Short term issues/decisions (1 minute)

  • xxx

Longer term issues/decisions: (9 minutes)

  • xxx

Upcoming tags/Next issues: (20 minutes)

  • xxx

Focused discussion (15 to 30 minutes...)

  • xxx

February 23rd, 2023

Agenda

CTSM Software wins for the week (5 minutes)

-- Sam L. was able to get the GULU branch updated to the latest and should be next tag! -- CGD Culture survey general report was released! Get back to us on questions/concerns... -- CGD "On the Spot" recognition https://docs.google.com/forms/d/e/1FAIpQLSdyb485HYb_fmhpnwrWMfddI_OTxOt10AngFR7kmd157aGBnA/viewform

FATES Update (10 minutes)

  • C balance checking at the column-scale on the FATES side? (grid-scale on the CTSM side)

Discussion items: (15 minutes)

  • We want to have the main part of this meeting an hour, and then a half hour for a given focused topic.
  • Will and Erik will review the agenda 15 minutes before each week
  • We want to be sure to cover weekly "wins", but be careful of time for meeting efficiency
  • Be careful of FYI type things that could be covered in an email
  • We have to be careful about talking about open positions. We want all qualified candidates to apply, and give them an equitable shot based on their ability. We can't give special preference to anyone, just because we work with them.
  • Please feel free to bring up ethical issues and have us talk about them as a group. If you don't feel comfortable bringing it up with the group, please bring it someone that will get it to Will.

Short term issues/decisions (1 minute)

  • Focussed discussion will be on conda (see below), who wants to join? Adrianna and Jackie?

Longer term issues/decisions: (9 minutes)

  • We have 91 open bugs. Let's make a concerted effort to address this. Can we do this in the next few months?

Upcoming tags/Next issues: (20 minutes)

Focussed discussion (on conda) (15 to 30 minutes...)

  • Erik will show what he's setup in py_env_create at this point

Structure of meetings

The last half-hour (10:30 - 11:00) can be used either for a pre-determined topic or set aside as a time that can be used if anything comes up that needs some longer discussion (especially if not everyone needs to be there).

Addressing open bugs

Suggestion of a bug-squashing party: take 2 weeks for a few of us to try to get through as many bugs as possible.

Some thoughts on this:

  • Consider having a daily stand-up during this period
  • Try to drop other things from our schedule (including user support) for this period
  • Good opportunity for pair programming, especially with newer SEs
  • Probably start the two-week period with a group triage

Keith: it would also be good to have a similar User's Guide update party... maybe not as fun, but needed.

  • Pair writing can also work well here: more fun, and gives essentially real-time review / editing.

Tentative scheduling: Do one of these in the first two weeks of April, and maybe another in July.

Regional subsetting & dask

Erik wasn't able to identify a single commit that could be backed out to remove dask.

Adrianna suggested that you can probably just swap out use of dask arrays for numpy arrays in a couple of places: Dask arrays are just numpy arrays with overhead.

Plan is still to remove the Dask stuff as long as Erik can find an easy way to do that.

What Erik has done so far is to make Dask optional in the conda environment. But we could just remove Dask entirely... feeling is that we should.

It seemed to Erik like the conda issues were coming not just from Dask but also from some plotting stuff for plotting meshes (matplotlib and cartopy). That plotting of meshes is really nice, but it adds some complexity to the python environment, so Erik is making this plotting optional.

We could put the plotting functionality in tools/contrib, and then not worry about supporting it.

  • We could have a plotting environment as a separate environment, or let users use their own python environment. Feeling is probably do the latter.

For some packages, we want / need to specify specific versions. e.g., for black, we want to be sure everyone is using the same version. For others, there can be some practical benefit to specifying a specific version.

Erik also proposes having a version of requirements that lists specific versions.

  • Bill's concern is that, if we're not testing it regularly, it could be a pain to maintain.
  • Erik wants to try adding it and see if it becomes painful or not.

February 16th, 2023 - side meeting on how to structure these meetings

See notes here: https://docs.google.com/document/d/1UkC26yS9Ij3iIRbem346wEvo9n14idnQ3XHauRZhEQI/edit#

February 16th, 2023

Agenda

CTSM Software wins for the week

  • LMWG was a big success.
    • Thanks for your presentations, Jackie, Erik, Ryan, & Sam.
    • Congrats to Keith for work with CLM-U, really impressive group of students presenting work at the meeting.
  • Erik made dev tag 118! (actually that was a small tag that Sam did)
  • Sam L's making progress on merging LULCC datasets for CTSM5.2
  • Sam R started in SE position

FATES Update

  • ELM making progress on C-based harvest with FATES (CLM is still behind on this) E3SM #5429
  • finished: return status checks in cohort deallocations #824
    • fixed problem with IBM and Cray compilers not working with FATES
    • Greg made a toy model to diagnose and solve this problem, also helped to read the compiler documentation
  • Ryan and Adrianna working on C and N balance checking between FATES and CTSM - what is the best way to do this?
  • Adrianna going to start implementing multi-column FATES

Short term issues/decisions

  • Will: Next round of coupled model testing will start in March 2023. Besides using the main ctsm5.1_dev tag are there other features we want to test out (e.g. CTSM5.2 surface dataset)?
  • Will: I saw that CAM is working on Cloud to ground lightning flash frequency passed to land model. I'm assuming we should be testing this out on the land side. As always... by whom and when? * Note, this may be more appropriate to discuss at a science meeting?
  • Erik: Can I get a subgroup to discuss conda?

Longer term issues/decisions:

Using new surface datasets

Sam L: To use the new surface datasets in CTSM 5.1, you need to make a couple of changes in the code:

  • Fix for nlevurb changes
  • A pft mask that is no longer present on the surface datasets

It sounds like Peter L is regenerating the PFT raw data again.

Sam R can post his changes that were needed.

Initial conditions for coupled model testing

Keith: for the CESM2 CMIP6 PI control run, we did a spinup with coupled model forcing.

Will: if the point of this is a smoke test to see if we can stop the Lab Sea from freezing, is it necessary to have this full coupled model-based land spinup?

Ndep status:

  • Scientifically, we should have essentially same ndep whether we get it from CAM or internally, if CAM is running without chemistry
  • Currently, for cplhist forcing, datm expects ndep, so CAM would need to provide that, unless we make some changes to turn that off.
  • We think no changes are needed in CTSM to get the ndep in from atm.

Cloud to ground lightning

See https://github.com/ESCOMP/CTSM/issues/1630 for some discussion on this.

conda environment

Erik: Two competing requirements:

  • It would be nice to only set the minimum versions needed (so can test with updated versions). But this doesn't guarantee that you're going to get a functional environment.
  • So there's also a rationale for specifying a specific set of versions.

Erik thought that conda lock would help with this. But in practice, this hasn't been working for Erik on either cheyenne or izumi.

Part of the complexity is dask. Erik suggests having a separate environment with dask. We might have to back out the dask stuff from this.

  • Jackie supports taking dask out of this regional subsetting.
  • Dask may be helpful for high-resolution grids, but since we aren't doing that now, it's not clear that dask is needed... even if things take an hour or so, that's probably okay as a one-time thing... and for most cases, it would probably take much less than an hour.
  • YAGNI

The problem with keeping this in is that it seems impractical to set up a working python environment right now.

Jackie has an environment that works currently. So for now we could use Jackie's environment if needed.

  • We wouldn't change the main CTSM python environment; instead, we'd have a note associated with the regional subsetting script about the need to create a particular environment.
  • Erik notes that this would make some testing not work. We could back out those tests temporarily.

Erik will look into how hard it would be to back out the dask stuff in order to determine the best path forward.

February 2nd, 2023

Agenda

CTSM Software wins for the week

  • Adrianna and I demonstrated the value of "Test Driven Development". You add a test, show that it fails, and then add the feature until the test passes. We did that and the test didn't fail -- which meant there was a problem with it.
  • It's ground hog day
  • Adrianna's NEON/FATES tag will be done before LMWG!
  • LMWG next week already!

FATES Update

Short term issues/decisions

  • Erik: Proposal for interface for subset_data to create mesh. Require --create-domain option (if you want a mask that isn't 1 everywhere). Domain files are no longer useful, but they are a way that we can get the mask. Will make it optional to do the plots. --create-mesh, --create-domain, will be required for --create-user-mods. So you either say "--create-domain" or something like "--land-everywhere", "--nomask", "--mask-active-everywhere". I'd like to tie it to the existing domain files for now as this will work, and eventually ESMF is going to have more tools for mesh files. Another way to do this is to use the mesh_modifier tool -- but still I need a valid mask for the region.

    • --create-domain only needed for regional runs that include ocean to create masks, otherwise this can be a --land-everywhere flag
    • Otherwise, this PR will make MCT obsolete (with the exception of the domain file that we're still using for a land mask).
    • This isn't ideas, but it seems like this is a short-term solution that ESMF will help support moving forward.
    • Erik feels like this is a more expedient way to run regional cases, without fixing bugs in the mesh capabilities from regional script that Negin made.
    • Erik will finish the regional PR and work on cleaning up the sparse grid capabilities is working too, see #1919 and #1731
    • Next tag will remove MCT testing from CTSM

  • Erik: Subtle limitation with the mesh maker right now is that it will only work on regular lat/lon grids. This is a problem for CTSM/WRF grids.

    • This will also be an ESMF task moving forward, but Sam Levis' seems to have a workaround using ncks, again see #1919
    • Erik and Sam will meet to discuss this further.

Longer term issues/decisions:

  • Erik: We've been talking about CESM grid aliases in CSEG because of mizuRoute. We plan to have mizuRoute grids spelled out for example: f09_f09_rHDMAlk_mg17. Is there still a need for tri-grids? And is there work where the atmosphere will run on a regular grid and CTSM on HRU's? We currently aren't testing tri-grids I think that's a problem.
    • Likely a science discussion, when do we want to run land on different grid than the atmosphere?
    • Erik will create an issue for this, but for now a simple fix would be to create a test where we run the atmosphere on SE and land on FV grid.
  • Erik: Some things from the FATES meetings. FATES PR board. The pre-meeting to set the agenda looks useful.

January 26th, 2023

Agenda

CTSM Software wins for the week

  • ctsm5.1.dev116! With PR's started back in July finally in!
  • Sam and Erik resolved a bunch of SLIM test failures!
  • Adrianna's pronunciation -- Ahh-drianna. Please correct us when we get it wrong. A workplace of respect and belonging means we get each other names right. This is especially important for people with unfamiliar names (people from foreign countries) or for example a transgender colleague who transitions at work.

FATES Update

Short term issues/decisions

  • Erik: We have a mapping problem with the mizuRoute lake grid. We are also at a point where I need to work on the coupler to pass lake Precip and ET.
  • Erik: mizuRoute fields to pass. When we pass lake Precip and ET, the QGWL field (which normally is: lake, wetland, and glacier) probably should be just glacier and wetland. Should I setup a new set of arrays for QGW or overuse QGWL for both purposes? It's likely clearer to have another set even though it will add additional logic.
  • Will: DOC production and export, (issues #1458 & #1216) introduce a number of CTSM, river, and ocean science opportunities and SE challenges. I don't really know how we handle this without additional support? Marius will present and the BGCWG & we can take it from there.
  • Will: Should we check in on the CTSM6 project board at this meeting periodically?

Longer term issues/decisions:

  • Erik: We've been talking about CESM grid aliases in CSEG because of mizuRoute. We plan to have mizuRoute grids spelled out for example: f09_f09_rHDMAlk_mg17. Is there still a need for tri-grids? And is there work where the atmosphere will run on a regular grid and CTSM on HRU's? We currently aren't testing tri-grids I think that's a problem.
  • Erik: Proposed a meeting with Bill/Jackie for next week to talk about meeting agendas
  • Erik: Some things from the FATES meetings. FATES PR board. The pre-meeting to set the agenda looks useful.

Passing fields to mizuRoute

Erik: When we pass lake Precip and ET, the QGWL field (which normally is: lake, wetland, and glacier) probably should be just glacier and wetland. Should I setup a new set of arrays for QGW or overuse QGWL for both purposes? It's likely clearer to have another set even though it will add additional logic.

Bill's gut feeling: Maybe easiest to use QGWL always, but sometimes the lake contribution is 0. This feels similar to what we do with glacier, where contributions to certain flux fields are 0 if we are / aren't running with dynamic glaciers.

DOC production and export

Erik: one concern is that mizuRoute currently can't handle tracers.

Periodic review of the CTSM6 project board

Probably about monthly is the right frequency for this.

January 19th, 2023

Agenda

CTSM Software wins for the week

  • LULCC Hackathon for LMWG on schedule and structure developing

FATES Update

  • CTSM PR expectations
  • How to reword 2a so as to not suggest precluding turn on FATERS by default? (Prescribing from data will always be "better")
  • 2a. DON'T degrade or break other parts of the model (i.e., do no harm)

Short term issues/decisions

  • Erik: What would we like to have done before the LMWG meeting?

Longer term issues/decisions:

  • Erik: We've been talking about CESM grid aliases in CSEG because of mizuRoute. We plan to have mizuRoute grids spelled out for example: f09_f09_rHDMAlk_mg17. Is there still a need for tri-grids? And is there work where the atmosphere will run on a regular grid and CTSM on HRU's? We currently aren't testing tri-grids I think that's a problem.

PR expectations

Will clarifies that https://github.com/ESCOMP/CTSM/wiki/CTSM-PR-Expectations comes from requests from the community for a better sense of what the PR process / expectations are. This isn't meant to be a significant change to our existing process / expectations, but more writing down our existing process / expectations.

How to word 2a, currently worded as "DON'T degrade or break other parts of the model (i.e., do no harm)"

  • We don't want things that substantially degrade the system, but in practice many changes will lead to degradations in some areas
  • Decision: we'll take out "degrade"

Is there anything we'd like to have done before the LMWG meeting?

Erik will try to wrap up some tags; suggests trying to get Adrianna's recent FATES branch in.

Sam R would like to have his crop calendar stuff done (though not critical that it be totally merged by the LMWG meeting).

Surface datasets for FATES with NEON

Erik suggests having a more descriptive name for these surface datasets like "16PFTs_mixed"... probably not having "FATES" in the name because this isn't really FATES-specific.

Will asks if there will be changes to the wrapper script.

  • Adrianna: yes: it will add flags to do this the new way; if not specified, it won't change how it was done before.

January 12th, 2023

Agenda

CTSM Software wins for the week

  • Adrianna is back!
  • Penelope is growing and on her way in the world. Future scientist or engineer?
  • Teagan is officially part of ESCOMP and CTSM on github

FATES Update

Short term issues/decisions

  • Erik: I removed Negin from all her assigned issues, and made a bunch of them a "next" to figure out how to handle. I want to suggest that we need a dedicated person for WRF-CTSM work and as such should note that WRF-CTSM is stopped until there is funding for that person.
  • Erik: Mariana's PR breaks the perl build-namelist to the point I think we need to dig up the idea of moving to python sooner than later.
  • Erik: Is run_neon.py --fates something hard to do?
  • Erik: What would we like to have done before the LMWG meeting?

Longer term issues/decisions:

  • Erik: In talking with Peter L. he thought we should have different columns for natural vegetation (primary and secondary forest, pasture, and grazeland). I think we should open up a discussion on this.
  • Land use transitions with FATES, can we make some progress on theIs run technical side of passing information between the HLM and FATES?
  • Erik: We've been talking about CESM grid aliases in CSEG because of mizuRoute. We plan to have mizuRoute grids spelled out for example: f09_f09_rHDMAlk_mg17. Is there still a need for tri-grids? And is there work where the atmosphere will run on a regular grid and CTSM on HRU's? We currently aren't testing tri-grids I think that's a problem.
  • Erik: I'd like to establish some "norms" on how we plan on handling PR's (and issues as appropriate) so expectations for all are set. This would be things like when a new PR comes in, how soon we will assign someone to review it, how soon we will first schedule it, etc. There needs to be some flexibility, but also accountability and expectations on both our side and the author. One way to do this would be for me to come up with a straw man, and then bring in a few others for feedback. After we have a small group we bring it to this group and then to TSS and then CTSM science group.
  • Erik: Things go from "that can be done anytime" to "needs to be done now" really quickly. One part of the problem is lack of transparency in priorities and sequencing of order (this thing needs to happen so X person can do Y which enables Z person to do ZZ). For our internal stuff maybe we can add a "Needed for:" attribute to issues and PR's, so we can start to track this. We have this for version milestones like CTSM5.2, maybe other things could use it.
  • Erik: It's hard for us as individuals or groups to keep track of more than 3 things at once. I want to suggest a workflow where we each individually have three things we are working on. I'm not going to do anything with that other than for myself. But, as a group, for this meeting I'd like to have three goals we examine each work and work towards. The three goals I suggest right now are: CTSM3, CTSM5.2, and CTSM/FATES-v1. We should adjust to the three things that most of this group is working on. For each of these goals then we want to have a way to visualize how much progress we are making on that goal, with metrics to see progress and how far we have to go. Being able to see that we are making progress on a goal is motivating, adds interest, and feels good for all. I propose that we also highlight blockers and needs for the three goals.
  • Erik: Suggestion for list for monthly meetings:
  • Go through status on goals
  • Go through project boards on each goal
  • Go through high priority project board
  • Go through high priority issues
  • What do we need to accomplish next month?
  • How did we do on this month?
  • What are long term issues/priorities we need to bring up in TSS and/or CTSM-science?
  • Erik: Suggestion for list of quarterly meetings:
  • Go through all PR's and update status
  • Go through issues that affect answers: label "bug impacts science"
  • Long term priorities/issues that we need to bring up in TSS/CTSM-science?

FATES update

Something the FATES group has been talking about is having a new dimension in the output for land use type.

  • PR 971 needs testing in CTSM discussion for new dimension (LU-type and LU-type by PFT) and integration after E3SM5106
  • In progress: 971 (C starvation) (Jessie); 964 (land-use-category history var) add new dimension for LU-type and LU-type by PFT (after E3SM5106)
  • PR prioritization
    • Add “refactor, reduce technical debt” to list for 1 CTSM PR expectations
    • Discussion - bug fixes highlight biases, we should allow for this possibility
    • Ryan and Greg provide significant time to testing CTSM-FATES for integration of PRs

WRF-CTSM work

Agreement that this will be back-burnered for now, since we don't have someone to work on these. Bill will look through these issues and decide which, if any, should be totally closed.

Conversion of perl build-namelist to python

Erik has some experience putting together python build-namelists now for mizuRoute and SLIM.

It would be great to leverage Alper's ParamGen tool, which CAM is moving to.

run_neon for FATES

Erik: would it be hard to add a --fates option to run_neon?

Adrianna: technically it's easy. The harder part is deciding what the defaults should be for FATES vs. non-FATES. For example, there are differences in whether you want a dominant PFT or all PFTs for a site: most FATES users would want a mix of PFTs.

Jackie: one problem with run_neon is that it's too easy to treat it as a black box; with FATES, we really want people to think about these things and not just go with the default option.

Will: the advantage of run_neon is that it simplifies things for new users. There's an option to set up the case and not run it; would it make sense to use run_neon to set things up initially and then let FATES users modify things from there?

Adrianna has tried to break this into pieces: first getting things functional at all; then we can come back and build on this later.

Organization of NEON usermods

Erik presented a way that we could avoid duplication of the settings for each site. The downside would be that there are 3 instances of directories for each site.

Another option would be to have multiple user mods directories, either via multiple things in --user-mods-dirs or having something picked up via the compset – but that only works easily if we can key off of the compset or running via a script that sets the multiple --user-mods-dirs for you (because it's a pain to require the user to do that).

Another option is Adrianna's original suggestion of having a variable in the includes; this probably wouldn't be too hard if this variable can be limited to something that keys off of the compset.

Feeling is: let's move forward with something, even if it isn't perfect or where we want to end up long-term. This could either be what Adrianna has now or what Erik is suggesting. Then we can revisit this once we have a 3rd case and so have a better sense of what might be needed.

Regional mesh subsetting PR

Erik: one issue is that the mesh class doesn't have tests. We ideally would have tests for this, as for other things.

Bill: agree that we want tests with things like this... but it's most efficient for the tests to be written at the same time as the code... it feels inefficient to go and try to add tests to this now, and this is likely to hold up this PR. So suggests deferring tests and just adding these if/when someone comes back to this code to change it. One other note is that we might be replacing this mesh stuff with a more general CESM-wide or ESMF-wide solution at some point.

Will suggests adding some documentation in this module mentioning that this is untested.

Retrospective on today's meeting

It may be that we should start with upcoming tags, since that's something important that all people want to be involved with.

We may want more deliberate thought about what makes it onto the agenda – including what piece of upcoming tags are vs. aren't discussed.

January 9th, 2023 - software standup

Possible ways for outside SE to contribute to CTSM

Bill: There is an outside SE (a Google SE) who wants to volunteer some time helping with CESM, particularly if there are some refactoring / code cleanup tasks that he can help with.

Some ideas:

  • Probably start with simple bfb stuff
  • Documentation: could find some area of the documentation that isn't well fleshed out; he could learn how to do the given thing then document it for others
  • Bigger project: Convert Peter's tool from C to Python
  • Bigger project: Refactoring photosynthesis

January 5th, 2023

Agenda

  • CTSM Software wins for the week

  • Hope everyone had a wonderful holiday break.

  • I thought the meeting on meetings with Beatrice was helpful

  • FATES Update

  • Short term issues/decisions

  • Longer term issues/decisions

  • Erik: I think we should have some project meetings for longer term priorities on a monthly and quarterly basis. We might invite more people to these meetings.

  • Information gathering that's needed

  • Brainstorming on new ideas

  • Go through issues: next, high priority

  • Go through upcoming tags

Recommendations from meeting with Beatrice

Some big recommendations

  • Try to keep this meeting to an hour
  • Try to move prioritization to either the monthly TSS meetings or a separate monthly-ish meeting with more people present, so that more people have a voice in this.