2024 Software Engineering meeting notes - NGEET/fates GitHub Wiki

May 6, 2024

Wins and Celebrations :tada:

New group using FATES: Tau Liu at Michigan Tech. Teaming up with Chonggang. Interested in calibration and regional runs.

Upcoming meetings and events :spiral_calendar:

Machine outages :desktop_computer:

  • CISL: May 7-8, derecho
  • NERSC: May 15, perlmutter

PR board status update

Integrated: CCH water xfer functions, LAI-by-patchage

Updates

  • Small fire refactor: ready to test
  • Allometry unit testing: testing discussion held on Friday. Minor org refactor to be done.
  • Land use v2: Walkthrough held. Break out land use alignment from disturbance TBD.
    • CTSM API 36: drafted. run mode combo checks discussed and updated
    • How do we test spin up for land use?
      • Similar to the fact that we don't have AD testing
      • Sam Rabin knows how to add a new test in run_sys_test. Greg will tag up with him later.
      • Decision: make issue for testing spin up, but don't hold this up. Add standard regression tests for other landuse v2 modes.
    • Deadline: integrated Friday May 24

New

  • Land Energy Balance Solve
    • what is a good absolute case? Look at distribution of conductance based on different values and come up with reasonable delta.
    • Marcos: Stomata have minimum when close so what about using residual as the absolute tolerance?
    • Ryan: check in with Gautam for E3SM-side reviewer and feedback

Latest issues and discussions of note

  • no new issues or discussions of note

Calibration progress update

Tutorial update

  • No new updates

April 29, 2024

Wins and Celebrations :tada:

  • NGEET Phase 3 proposal submitted
  • First FATES newsletter sent out
  • ELM User's guide PR with connecting FATES documentation

Upcoming meetings and events :spiral_calendar:

Machine outages :desktop_computer:

  • CISL: none planned
  • NERSC: May 15, perlmutter

PR board status update

Latest issues and discussions of note

  • spin-up speed-up
    • Dan Ricutto might now
    • E3SM specific. Loads stuff into memory.
  • growth respiration calc
    • Initial solution consideration (Marcos): calculate on a daily basis and apply with each timestep
      • Need to have code to account for edge cases given if function based on previous day's value
    • We want to get this done prior to the nocomp calibration release
    • How does this apply with HLMs (big leaf)?

Calibration progress update

  • Adrianna: Daniel and Lydia discussion about setting up emulator. How much do we need for a good set for each pft?
  • Rosie: Rerunning one at a time nocomp ensemble with latest main branch (daylength + alt vert scaling).
    • Charlie: High seed rain clarification: Rosie wants to dig into this a bit more, ideally with fully spun up case.
    • Is there a danger in including seed rain in an ensemble for global calibration?
    • Rosie will make global map of seed rain impact

Tutorial update

  • Logistics: Organizing travel

Walk-on topics

  • Should we re-review this meeting time?
    • Goverance meeting?

April 22, 2024

Wins and Celebrations :tada:

Upcoming meetings and events :spiral_calendar:

  • FATES modeling meeting: April 25, Huanyuan Zhang
    • Jessie got confirmation for this week
    • Marcos will need someone to host May 23

Newsletter discussion

  • What sort of topics would people like to see?
    • Add request for fates users
  • How to generate content?
    • Asynchoronous update + in meeting review
    • Dump "wins" from these meetings into the document live
  • Ryan to setup calendar reminder for sending it out every quarter
  • Rosie: Post-doc position - Rosie

Metrics questions

  • How many countries and institutions use FATES?
    • Rosie has a pretty good handle on this. She will send out slide with this information.
      • US universities is an area of uncertainty.
      • Tutorial application could be used for metrics in this regard.
  • How many members do we have?

Machine outages :desktop_computer:

  • CISL: none planned
  • NERSC: May 15, perlmutter

PR board status update

  • API 35 integration and PR1128 is imminant
  • API 33 integration needs confirmation of developer tests passing
  • Landuse v2 walkthrough scheduling
    • Setup walkthrough meeting next Thursday in the timeslot for fates modeling
  • Ryan to reach out to hydro folks for final review
  • Adrianna looking for reviews for unit testing and fire refactor
    • Unit testing needs cime and shr
    • Greg and Charlie to review

Latest issues and discussions of note

  • Minor cleanup issue submitted by Sam Rabin

Calibration progress update

  • Rosie: shared parameter sensitivity results

Tutorial update

  • No new update

Walk-on topics

  • Carbon budget issue - DBEN - Jessie
  • Post-doc position - Rosie

April 15, 2024

Cancelled

April 8, 2024

Wins and Celebrations :tada:

  • Eclipse today! :sun_with_face: :new_moon:
  • Perlmutter speed up trick: DVS in read-only mode (see Jessie's email)
  • Jennifer's birthday

Upcoming meetings and events :spiral_calendar:

Machine outages :desktop_computer:

  • CISL: none planned
  • NERSC: 17 Apr, perlmutter

PR board status update

  • New allometric modes: Greg testing to check b4b issues Ryan seeing. Test reverting code in EDCanopyStructureMod
  • CCH water xfer functions: CCH hydro improvements coming along. Ryan now tracking down issue with non-debug hydro resulting in carbon balance errors.

Latest issues and discussions of note

Calibration progress update

  • Rosie: Adrianna and Rosie discussing approach for next steps using Adrianna's pft files (one-at-a-time no comp simulations)
    • Rosie to integrate changes from Jessie's vert scaling PR as well
    • Workflow discussion
      • Jupyter notebooks are not great from a version control perspective, but nicely steps through ensembles updates

Tutorial update

  • No new update

Hydro / SP mode compatibility (Ryan)

  • Ryan noting that we need to add new features to make hydro and sp mode work together.
    • As such, we shouldn't consider failure in the mode to be a bug, but a feature not yet supported.
    • Greg to take the action to add graceful failure with both modes in API35 and API34 ctsm and e3sm PRs respectively.

Walk-on topics

  • Rosie: Fire BVOC emissions
    • Getting MEGAN to work with FATES: https://github.com/NorESMhub/CTSM/issues/36
      • Keeping fire emission factors in fates parameter file and push calculations in FATES to HLM
      • Rosie currently has a new parameter to map from FATES to MEGAN. Rosie doesn't like that this makes the param file not HLM agnostic.
  • Small conference rooms available in B74 (314 and 202)

April 1st, 2024

Wins and Celebrations :tada:

  • Fire trunk fuel bug fix
  • Allometry parameters are in!
  • Hydro restart bug fix
  • FATES tutorial application notices sent out. We've gotten confirmation back from most of them.

Upcoming meetings and events :spiral_calendar:

Machine outages :desktop_computer:

  • CISL: 2-3 Apr, derecho
  • NERSC: 17 Apr, perlmutter

PR board status update

Latest issues and discussions of note

  • fire trunk issue: fixed per #1180
    • Discussion of impacts to current work.
      • Team is following up with those that were impacted.
    • Adrianna to run coarse global 100 year run to get some assessment of impact. She will update the issue with results and close it out.
    • Future refactor is to not include trunk loading in fuel class
  • SP restart crash: since there is no mort in SP mode, we will avoid these in fix

Calibration progress update

  • Jessie: high biomass issue may be alleviated by SP parameters from Adrianna's work. Also recent fire fix may help as well.
  • Adrianna: Planning on rerunning with new pft composition to address sparse grid limitations

Tutorial update

see wins

Walk-on topics

  • Adrianna: Testing stand structure faster (testing initialization)
  • SSH security heads up: backdoor found inxz-utils version 5.6.0
    • Fedora Linux 41 users should check security alert notice

March 25, 2024

Wins and Celebrations :tada:

Upcoming meetings and events :spiral_calendar:

Machine outages :desktop_computer:

  • CISL: 2-3 Apr, derecho
  • NERSC: 17 Apr, perlmutter

PR board status update

Fire refactor overview

Latest issues and discussions of note

Improving PR integration workflow

  • Brainstorming ideas for reducing PR backlogs
  • Reviews
    • Hold more regular code walk-throughs for large PRs
    • Break-up and stage large PRs if possible
  • Testing
  • Other logistics
    • On-the-fly default parameter file build
    • parameter patch file auto-build/checking

Calibration progress update

deferred

Tutorial update

deferred

March 18, 2024

Cancelled

March 11, 2024

Wins and Celebrations :tada:

Upcoming meetings and events :spiral_calendar:

Machine outages :desktop_computer:

  • CISL: none planned
  • NERSC: Wed 20 Mar, perlmutter
  • OLCF: Tue 19 Mar, center-wide

PR board status update

  • CCH water xfer functions: resolves #1163, but now seeing 1D hydro failure
    • Marcos/Ryan: capping psi_min and th_min discussion?
      • Discussion with Jenny last week (water balance conservation issue).
        • Ryan: Use linear model below zero similar to Van Genukten?
        • More discussion needed
    • Greg to track down regression test failure

Latest issues and discussions of note

Calibration progress update

  • Rosie: discussion of disconnect between tuning and diagnostics packages for land

Fire emission coupling hackathon (Rosie)

  • Goal: get fire emissions out of fates and into atmosphere model
  • Calculate biomass burned and pass to HLM to calculate emissions or fates calculate emissions directly
    • emissions are pft specific suggesting we should have fates calculate directly
      • would result in duplication of emissions factors from HLM
  • Charlie: is emission a function of fire "type" (i.e. anthro versus wildfire)
    • Rosie: yes, but pft is a proxy of this
  • Ryan: discussion or issue?

Hydaulic failure discussion (Chonggang)

  • Option: when leaf water potential high, zero stomatal conductance and not request water
    • May improve numerical stability
    • Chonggang hoping to attempt this at the end of the month
  • Ryan: preference is to implement linear model between
    • Alternate (short term?): run case at much shorter time steps to get through snags
  • Charlie: Develop dry condition test failures (e.g. single site at Atacama)
    • Ryan: functional unit testing
      • satphen + hydro?
    • Greg: being able to parameterize functional unit tests would be great

Tutorial update

  • deferred

March 4, 2024

Wins and Celebrations :tada:

Upcoming meetings and events :spiral_calendar:

Machine outages :desktop_computer:

  • CISL: Tue 5 Mar, derecho
  • NERSC: Wed 20 Mar, perlmutter
  • OLCF: Tue 19 Mar, center-wide

PR board status update

  • Stuck:
  • Update:
  • Discussion:
    • Vertical scaling: Discuss email thread on clarifying scalar definitions and migration to parameter file
    • Action:
      • Ryan to create new issue for discussion about sla scaling in the biophyscial rates routine
      • Jessie: add parameters Ryan suggested to PR to tee up for future implementation without needing to worry about future API update.

Latest issues and discussions of note

  • No new issues or updates

Calibration progress update

  • No new issues or updates

Global stomatal conductances (Ryan)

  • Making coupling HLM-FATES faster. Honing in on the right stomatal conductance.
    • Noticing how wide range of values are globally. Why values in the sahara (full fates)?
    • Ryan will run an sp mode and share results

Tutorial update

Walk-on topics

  • Rosie: mini hackathon next Thursday/Friday to tackle the fire emissions coupling.
    • Add to discussion agenda for next Monday.

February 26, 2024

Upcoming meetings and events

FATES software meeting check-in

  • Is there interest in moving this meeting to an every-other week cadence?
    • Consensus: those present like the weekly cadence and the reporting back of PRs and issue status.
    • Greg: What about the notes format? Would people prefer to engage with the notes in another format?
      • Consensus: those present like the transparency of the wiki format and that the notes are co-located on the repo with everything else (issues, discussion, code, etc.)

Machine outages

  • CISL: none planned
  • NERSC: 20 Mar, perlmutter
  • OLCF: none planned

PR board status update

Updates:

Latest issues and discussions of note

  • #1163 and #1165 addressed by PRs #1164 and #1163, respectively

Calibration progress update

  • Jessie: finishing up respiration work; will turn into maniscript in near future.
    • Considering adoption of Adrianna's ML method for next iteration (post SP-mode calibration updates)
    • Ryan: stem respiration for calibration?
      • Jessie: Not currently. Kolby keen to investigate this for next project phase. Discussions about field-to-model integration details on going.

Tutorial update

  • Jennifer: what's the accessibility goal? Will we explain how to setup the container, etc?
    • Jessie: Ultimate goal is to make this available all online with a very simple BCI method.
      • Greg: We also have a tutorial setup guide repo in development to host a readthedocs style walkthrough of setting up on a laptop.
    • Greg: OLMT experience? We might use or compare the single site subsetting function for pre-processing.
      • Jennifer: Yes, two years ago. Current attempts to use were hampered however.
      • Ryan: Running pre-process all at once?
        • Jessie: yes, we could use the sparse grid NCL method.

February 12, 2024

Upcoming meetings and events

Machine outages

  • CISL: none planned
  • NERSC: 21 Feb, perlmutter
  • OLCF: none planned

PR board status update

Latest issues and discussions of note

FATES speed audit update (Ryan)

  • Can we avoid leaf layer loop: Trimming routine forces us to iterate over leaf layer depth
  • Current plan: investigate Ci iteration cap and threshold. Update as necessary along with remembering the previous value.
  • Initial iteration analysis update suggests that we often hit the iteration cap.
    • Initial iteration testing with ST3 mode (audit was using full dynamics however).
  • ED2 experience: What about trying Newton method using the derivitives from reduced analytical form? Review the ED2 approach and share with team.
  • Investigate impact of making photosynthesis loosely coupled?

Calibration progress update

  • Adrianna: CLM surface data does not seem to quite match ILAMB observation. This is causing issues with gridcell to gridcell comparison.
    • Ask Nate Collier for help.
  • Calibration team to review Adrianna's brute-force method and will review selected parameter ranges.

February 5, 2024

Upcoming meetings and events

Machine outages

  • CISL: 5-7 Feb, derecho
  • NERSC: 21 Feb, perlmutter
  • OLCF: 6 Feb, frontier

PR board status update

Latest issues and discussions of note

Calibration progress update

January 29, 2024

Upcoming meetings and events

Machine outages

  • CISL: Mon 5 - Wed 7 Feb, derecho
  • NERSC: none planned
  • OLCF: none planned

PR board status update

Latest issues and discussions of note

  • LU data tool fails on derecho: probably taking too much memory for login node. This will need to be addressed eventually for CLM LU build on the fly.
    • This came up in the context of checking ctsm python environment on derecho, due to a dependency issue for xesmf.
    • Erik/Charlie: test this on the develop queue

Calibration progress update

  • Adrianna: Ran Ball-Berry without temp acclimation. Observational mean is more easily achieved, but emulator does a worse job.
    • Temperture acclimation without very low vcmax
      • Charlie: Claire Zarackas is finding something similar in the absence of hydraulics. So there is a scientific rationale for excluding temp acclimation.
  • Jessie: met with Ryan to talk about carbon budget issue; add recruitment carbon flux. Carbon budget closes now.
    • Not sure if recruitments that die in the same timestep are showing up in the diagnostics

Walk-on topics

  • Rosie: Question: Matvey fates model focus? Take over Marius work? Where does it fall in the priority?
    • Jennifer: arctic-fire stuff coming up prior to arctic-hydro. Maybe hardening?
    • Greg: This is back on API 17, so bringing it up before it gets too stale would be good
    • Charlie/Ryan/Rosie: recent hydro issues do suggest that addressing hardening is a good idea.
    • Matvey: only calibrated to 2 local sites; will it work globally?

January 22, 2024

Upcoming meetings and events

Machine outages

  • CISL: Mon 5 - Wed 7 Feb, derecho
  • NERSC: none planned
  • OLCF: none planned

PR board status update

Building parameter files on the fly

  • Ryan to look at moving this forward this week

Latest issues and discussions of note

  • hydro dry region failure: Marcos has been working with Jenny on this. Recommendation is that she not trace the issue since its out of scope for her work.
    • Ryan proposing to add a fail safe for extreme outlier cases to allow fates run to complete. Still need to make sure this is actually an outlier.

Calibration progress update

  • Calibration branch
    • Jessie still needs this branch until 2-stream gets to E3SM (probably a few weeks at this point). The calibration branch is pretty close to FATES main for her purposes, so no need to merge main into calibration branch.

Walk-on topics

  • Erik asking about status of Marius' frost branch

January 15, 2024

No meeting due to lab holiday

January 8, 2024

Upcoming meetings and events

Machine outages

  • CISL: Mon 5 - Wed 7 Feb, derecho
  • NERSC: Wed 17 Jan, perlmutter
  • OLCF: none planned

PR board status update

Latest issues and discussions of note

Calibration progress update

  • Jessie presented latest results: GPP and biomass bias improvements

January 1, 2024

No meeting due to lab holiday