2024 Software Engineering meeting notes - NGEET/fates GitHub Wiki

December 16, 2024

Wins and Whimsy :tada:

  • Former tutorial student requested to use fates tutorial for Ghent University vegetation moduleing course
    • Jessie and Greg to reply. Add Ryan and Adrianna to the reply as they are interested in the course content as well.

Machine status and notices :desktop_computer:

  • CISL: none planned
  • NERSC: Wednesday 18 December

Upcoming meetings and events :spiral_calendar:

PR board status update

  • Greg and Ryan to schedule longer outlook planning session for PR board in the new year

Integrated

Update

New

  • parameter switch migration
    • To be coordinated with ESCOMP/CTSM#2904
      • CTSM PRs waiting on high priority PR ahead of this
    • E3SM-side pull request TBD
    • CK: Should we update defaults at the same time? E.g. Medlyn.
      • RK: Jenny has update to default.
      • GL: We can, although there is an opportunity to bring this in on the ctsm b4b-dev branch if we don't
        • That said, doing so is dependent on some other ctsm queue factors
    • CK: how should we handle real number parameters that are switch dependent? How to make the user aware?
      • Differences between defaults for the host land models will eventually exist in parameter file
    • RK: This is a short-term solution. Should we accelerate moving to migrating parameter file to xml?
      • This would allow us to nest and check parameters based on the switch
      • Consensus is moves this forward as it solves current testing issues. Conduct
  • Don't zero patch%fuel%frac_burnt: needs review
    • Fixes issue #1301
    • CK: Fuel amount conditional on patch age. Adrianna to review.
  • Fix cmake for functional tests
    • AF: note that if you are running with latest ctsm tag, functional tests will break without this PR

Latest issues and discussions of note

Calibration progress update

  • deferred for walk on topic

Walk-on topic

  • Rosie: previewed FATES diagnostic work
    • TVEG issue discussion: sp mode looks ok, nocomp is too low.
      • RF: running with default parameters
      • RK: hlm_hio_ignore_val should not be averaged in
      • CK: make a map of equivalent host land model to see how it compares. Should align with fates.

December 09, 2024

Cancelled due to AGU

December 02, 2024

Wins and Whimsy :tada:

Machine status and notices :desktop_computer:

  • CISL: none planned
  • NERSC: Wednesday 18 December

Upcoming meetings and events :spiral_calendar:

PR board status update

Integrated

Update

Latest issues and discussions of note

  • inventory restart question (fates-tutorial#64)
    • Greg and Ryan to ask clarifying questions
    • Suggest toggling the hard-coded switch to get more diagnostics
  • SP mode + finidat case failure (Adrianna)
    • Height coming from hlm is zero for some reason in sp mode when running a case copied from a previous case
      • Restarting from original case does not have this problem
    • Adrianna to create an issue (if necessary)

Calibration progress update

  • Adrianna to send python code to Jessie for sparse grid mapping

Patch allocation and boundary arrays discussion (Ryan)

Walk-on topic

November 25, 2024

Wins and Whimsy :tada:

Machine status and notices :desktop_computer:

  • CISL: none planned
  • NERSC: Wednesday 18 December

Upcoming meetings and events :spiral_calendar:

PR board status update

Update

New

Latest issues and discussions of note

Calibration progress update

  • Adrianna: Discussion of recent results

Walk-on topic

November 18, 2024

Wins and Whimsy :tada:

  • Adrianna's first FATES merge and tag.
  • NorESM merged MEGAN-FATES update to their ctsm branch. Will be merged back to ESCOMP ctsm.

Machine status and notices :desktop_computer:

  • CISL: none planned
  • NERSC: Wednesday 18 December

Upcoming meetings and events :spiral_calendar:

Activity planning board

  • Initial thoughts
    • Advertise board at modelling meeting and in newletter
    • Inviting people to contribute?
      • Greg to generate google form for people to have their project added
  • Archiving and converting items to issues? Consensus: yes

PR board status update

Integrated

Updates

New

Latest issues and discussions of note

Calibration progress update

  • Rosie: making updates to ensemble

Walk-on topic

November 11, 2024

Wins and Whimsy :tada:

Machine status and notices :desktop_computer:

  • CISL: none planned
  • NERSC: Wednesday 13 November

Upcoming meetings and events :spiral_calendar:

Forward looking project board (Charlie)

  • This came out of a discussion between Charlie and Adrianna.
  • Both forward-looking (planning) and forward-facing (outreach)
    • A lot of our boards are for current efforts (e.g. PR board, Landuse board)
    • We want community to know what we are doing and know how to reach us for collabs
    • Jessie suggested reviewing this at the modeling meeting to encourage used
  • Allowing people to add their own items
  • Greg to sketch out the board which we'll review in this meeting next Monday

PR board status update

Updates

New

Latest issues and discussions of note

Calibration progress update

  • No updates

November 04, 2024

Wins and Whimsy :tada:

Machine status and notices :desktop_computer:

  • CISL: Tuesday 5 November
  • NERSC: Wednesday 13 November

Upcoming meetings and events :spiral_calendar:

PR board status update

Integrated

Updates

Latest issues and discussions of note

  • Adrianna: imaginary quadratic solve issue
  • Rosie: finidat issue
  • Rosie: testing + compset concerns from Marianna

Calibration progress update

Walk-on topics

October 28, 2024

Wins and Celebrations :tada:

Machine status and notices :desktop_computer:

  • CISL: Tuesday 5 November
  • NERSC: Wednesday 13 November

Upcoming meetings and events :spiral_calendar:

PR board status update

Integrated

Updates

New

Latest issues and discussions of note

  • sapflow bug (#1266): Discrepency between FATES_SAPFLOW_SZPF and FATES_SAPFLOW
    • Charlie/Marcos: Units are incorrect
  • Branch runs failure (#1271): Sam tested and found that branch runs fail for FATES. Tested using ERI prefix, for which no fates test exists.

Calibration progress update

  • Adrianna: GPP improvement globally (GBAF, FLUXCOM, WECAN). Slightly under calculating in the amazon now. Much better everywhere else including Africa.

Walk-on topics

October 21, 2024

Wins and Celebrations :tada:

Machine status and notices :desktop_computer:

Upcoming meetings and events :spiral_calendar:

PR board status update

Integrated

Updates

Low relative humidity discontinuity discussion

  • Discovered via Demo of Leaf Biophys Unit Tests
    • Are the discontinuities due to singularities in stomatal conductance calculation?
    • Marcos has seen this in ED2 as well and addressed it.
      • Dynamically make sure that the initial guesses don't straddle the singularity.
  • Only an issue with Ball-Berry. Move to medlyn as the default?
    • Adrianna: Melyn has a correction that doesn't allow humidity to get that low (cap on VPDs between leaf and atm, not atm itself).
      • CLM apparently fixed the issue and removed the cap.
        • Ryan removed the caps for medyln and didn't see the issue reappear.
      • That said Adrianna is seeing convergence errors.
    • Ryan and Marcos to setup a meeting to discuss

Latest issues and discussions of note

  • Grass sapwood allometry hydro issue (#1254): Xiulin to create PR to address this soon.
  • Biomass outputs updating for non-new cohort only
    • Ryan: if its a state should be included, but if its a flux it shouldn't be used. We need to do an audit of Sam's list and what is/isn't a flux.
      • Marcos: Issue is more if its per plant rather than per area.
    • Future refactor: splitting history interface mode between fluxes and states
    • Immediate work: audit the list and create PR to correct. This should be based on Sam's PR and come in afterwards.
  • FATES simluation in desert environs: Rosie answered. Cross-post low relative humidity issue here as well.

Calibration progress update

  • Adrianna: getting results for co-dominant

Walk-on topics

  • Sam: distinction between subroutine hist dyn 1 and 2?
    • Adrianna: suggests renaming the subroutines to be more descriptive

October 14, 2024

Wins and Celebrations :tada:

Machine status and notices :desktop_computer:

  • CISL: Tuesday 5 November
  • NERSC: Wednesday 16 Oct

Upcoming meetings and events :spiral_calendar:

PR board status update

Integrated

Updates

Latest issues and discussions of note

Calibration progress update

Walk-on topics

October 07, 2024

Wins and Celebrations :tada:

Machine status and notices :desktop_computer:

  • CISL: none planned
  • NERSC: Oct 16, Perlmutter

Upcoming meetings and events :spiral_calendar:

PR board status update

Integrated

Updates

Latest issues and discussions of note

Grass sapwood allometry hydro issue (#1254)

Calibration progress update

  • Charlie: Seed rain impact on spin up discussion (wrt nocomp calibration)
    • Seeds may be supporting a population of plants that are constantly dying
    • Should we be making some metric/ecosystem benchmarks to make sure that certain pfts are being represented
      • It appears that die-off is mostly in grasses
      • Investigate using Xiulin's new grass allometry
      • Charlie to update with more detailed maps showing the differences
      • Maybe test seed dispersal

Walk-on topics

  • Rosie: FATES reading 78 pft issue
    • NorESM-FATES needs 16pft file for spectral element simulations as well
    • Discussion triggered by issue (#2810) testing showing that we needed a file for ctsm5.3.0.
      • Sam L and Mariana have created to address.

September 30, 2024

Upcoming meetings and events :spiral_calendar:

PR board status update

Updates

Latest issues and discussions of note

1250

Calibration progress update

Walk-on topics

September 23, 2024

Machine status and notices :desktop_computer:

  • CISL: none planned
  • NERSC: Oct 16, Perlmutter

Upcoming meetings and events :spiral_calendar:

PR board status update

Updates

  • Arctic shrub PFT additions (#1236): queueing and regression testing
    • Jennifer: running an additional smoke test to conduct some difference mapping
    • updates are a result of observation data and FASST optimatization
    • Ryan: coordinating with Adrianna's sp mode update
      • Adrianna current work doesn't include bl ever extra trop, and broad decid extra trop
        • Since so much is changing with sp mode calibration inputs (e.g. gswp3 -> cru), might as well update this and redo sp calibration
    • Jennifer: Drive to get this in is to meet NGEE-A milestone
    • Greg: start regression testing to get ahead of any other issues since API updates are coming up soon in queue
      • Greg to start e3sm API update
  • Time integrated flux diag (#1217): under review
    • Jessie to add review later today
  • new patch insertion method (#1155): under review, Ryan may implement recommended changes for him
  • Spitfire fuel calc refactor (#1247)
    • Adrianna: question about historical context for zero fuel code comment
      • Adrianna will update this to allow for zero fuel during her refactor

Latest issues and discussions of note

No new issues or discussions of note

Newsletter feedback

  • Marcos: should we be advertizing FATES presenations as well?
    • Ryan to add call for people to provide highlights for a presentation and papers
    • Marcos to create google form to provide consistent input results
    • Goal: get your edits in by Friday. Ryan will send out then.

Calibration progress update

  • Jessie, Rosie, and Adrianna to meet on Wednesday
  • Linea and Adrianna to create repo for calibration

Walk-on topics

September 16, 2024

Wins and Celebrations :tada:

  • 2024 FATES Tutorial event was a success!

Machine status and notices :desktop_computer:

  • CISL: Sep 16-19, Derecho
  • NERSC: Oct 16, Perlmutter

Upcoming meetings and events :spiral_calendar:

  • FATES calibration coordination meeting: CLM meeting, September 19
  • FATES modeling meeting: September 26, TBD
    • Ask Steven, who will be here at LBL, if he'd like to present in October
    • Bianca Rius for a model comparison. Marcos to reach out.
    • Reach out to Yanyan when Rosie is back

Tutorial Retrospective

  • Three attendees have signed up on the FATES model google group so far
  • Generated a couple issues and started up some PRs during the tutorial

Initial feedback

  • Just under half responded via survey so far: mostly positive feedback, some constructive critism
  • Software: Docker setup pre-tutorial could use improvement; more introductory material on working in the terminal
  • Presentations: Slow down speech cadence during presentations and spend a little more time during intro talks
  • Balance of time between hands-on practice and presentations: slight bias towards more presentations with examples of use

FATES team feedback

  • Ryan: Presentations as breaking up the day
  • Jessie: more presentations on different processes in fates (guided examples)
  • Ryan: make it more clear that they can take breaks when they want
  • Sam (via Jessie): have a section on debugging. E.g. where are the log files?

PR board status update

Updates

New

Latest issues and discussions of note

Calibration progress update

  • Adrianna: no major updates due to machine outage. Coordination meeting coming up this week.

Walk-on topics

  • Sam: refactoring history PR work
    • Having trouble reproducing result from the baseline. Needs a second pair of eyes to review.
      • Ryan to tag up with Sam.

September 9, 2024

Cancelled due to FATES Tutorial event

September 2, 2024

Cancelled due to Federal Holiday

August 26, 2024

Wins and Celebrations :tada:

Machine status and notices :desktop_computer:

  • CISL: Sep 16-19, Derecho
  • NERSC: Sep 11, Perlmutter

Upcoming meetings and events :spiral_calendar:

  • FATES modeling meeting: September 26, return to regular cadence
    • Charlie added Yanyan Cheng recent paper. Ask Rosie about having them present paper.
    • Jessie schedule Anna Marie in November. Waiting for confirmation.
  • FATES calibration coordination meeting: CLM meeting, September 19
  • FATES software meeting: move to every other week cadence?
    • Informal poll: No strong feelings. Meeting is well run and meeting weekly works well.
      • Check in with Rosie and others about finding a better time for the future.

PR board status update

Updates

  • Fire Weather Refactor: Updates made and retested. Results as expected.
    • To be integrated next. No need to wait for CTSM API updates.
  • Arctic shrub PFT additions: Should this update the default or have a new cdl file? If not, how do we test this?
    • Making default cons: not tested pan-arcticly (one arctic site only), would complicated calibration efforts (?)
      • Adrianna: there have been multiple parameter file updates during calibration, so not that big of a concern for SP calibration work
      • Check in with Rosie about non-comp impact
    • Multiple parameter files? Consensus is no; either update default or just provide xml for this PR.
      • Testing with xml only option: we could test this using the parameter auto-gen workflow
      • TODO: run long global 4x5 smoke test. If results look good, make this default.
  • CTSM API 36.1 update: Waiting on higher priority ctsm-side PRs to meet CTSM5.3.0 tag

New

Latest issues and discussions of note

  • CanopyStructure third canopy layer
    • Added new clause is causing issues due to tolerance. Some version of check should be performed, but not sure what tolerance.
      • Ryan: imperfect plasticity would help this, but radiation scattering makes that difficult.
    • Charlie: leave open; needs more investigation.
  • EDPFTVarcon merging: Ryan will address this.
  • FATES inventory pft error: Greg to reach out to author to ask for files
  • Charlie's Restart Error: under investigation; Ryan reproduced error
    • Ryan: Not all of the params the drive 2-stream have reasonable values on the restart. Adding error codes to exit back into calling subroutine.

Calibration progress update

Adrianna: Shared recent results on broadleaf evergreen to get feedback

  • Atkin and stomatal conductance highly correlated.
    • Jessie: is this related to odd behavior between Atkin and stomatal conductance that Polly emailed about?
  • Workflow only took an hour

Tutorial update

Walk-on topics

August 19, 2024

Wins and Celebrations :tada:

Machine status and notices :desktop_computer:

  • CISL: None scheduled
  • NERSC: Aug 27, Perlmutter

Upcoming meetings and events :spiral_calendar:

  • FATES modeling meeting: September 26, return to regular cadence
    • Stephenn deHertog to be here for a month after the tutorial. Marcos to invite him to talk on this date.
  • FATES calibration coordination meeting: CLM meeting, September 5
    • Goal: coordinate existing effort and timelines

PR board status update

Integrated

Updates

New

Future

  • Chonggang: Beetle code forthcoming
  • Chonggang: Water potential based on salinity just getting started
    • Rosie: Has done experimental water potential work with mangroves
  • Sam: Prioritizing escaped fire work
    • Sam to work on replicating Rosie's fire work (3 pfts)

Latest issues and discussions of note

  • 2-stream restart error: testing on main with Charlie's minimum script is passing
    • Charlie: Issue has to do with grazing. He's confirmed that removing grazing alleviates the issue he was seeing.
  • Charlie: canopy structure issue (to be )
    • There is a check missing that needs added to make sure that the canopy is full before adding to lower canopy layers
    • Immediate fix is to change order of promotion from lower to upper
    • Whats the reasonable tolerance level on check?
      • Rosie: check the largest error for crash
    • Prioritization of conservation values (dbh, mass, etc) resulting in conflicts
    • Marcos: ED2 doesn't conserve DBH and back-calculates. FATES is more complicated.

Calibration progress update

  • Adrianna: No major updates. Script for dominant and co-dominant grid working.
  • Greg: documentation of process and tools used? Maybe this should be a discussion point for the coordination meeting?

Tutorial update

  • Beta-testing started. Greg conducting OS testing on Windows. Jessie receiving inventory coordinates to generate surface/domain data subsets.

Walk-on topics

  • Rosie: AD spin-up use recently (in CLM-FATES)? Seeing crashes.
    • Charlie: should work, although had to comment out some phosphorous code in ELM for something specific
    • Adrianna suggested looking in CESM forum. She recalls seeing some issue posted there.
  • Chonggang: Topounits with FATES?
    • They should theoretically work together

August 12, 2024

Wins and Celebrations :tada:

Machine status and notices :desktop_computer:

  • CISL: Derecho maintenance canceled
  • NERSC: Aug 27, Perlmutter

Upcoming meetings and events :spiral_calendar:

PR board status update

Latest issues and discussions of note

Calibration progress update

  • Rosie: objective SP mode calibration work shifting from derecho to betsy (NorESM machine)
  • Adrianna: workflow using PFT grids for SP mode targeted to be finished soon.

Tutorial update

  • Beta-testing scheduled to start this week. Content continuing to be expanded on.

Walk-on topics

  • Ryan: Working on refactoring photosythesis module and adding unit tests

August 5, 2024

Wins and Celebrations :tada:

Machine status and notices :desktop_computer:

  • CISL: Derecho maintenance canceled
  • NERSC: Aug 27, Perlmutter

Upcoming meetings and events :spiral_calendar:

PR board status update

Latest issues and discussions of note

Calibration progress update

Tutorial update

August 5, 2024

Wins and Celebrations :tada:

Machine status and notices :desktop_computer:

  • CISL: August 12-15
  • NERSC: July 31, Perlmutter

July 29, 2024

Wins and Celebrations :tada:

  • Land use v2 E3SM API integrated

Machine status and notices :desktop_computer:

  • CISL: August 12-15
  • NERSC: July 31, Perlmutter

Lawrencium support

Lawrencium moved from CentOS to Rocky8 and the cime configuration to support this machine is broken

  • Do we want to prioritize an effort to fix this in the long term?

Upcoming meetings and events :spiral_calendar:

PR board status update

Use of fates synchronized branches with CTSM (not master)

  • Motivation: FATES development team can merge in PRs that are synchronized with CTSM on our own schedule, allows users to have a CTSM branch that is always compatible with FATES master
  • Questions:
    • Should we have a single branch (fates-latest)? Or a branch for each expected tag?
    • What frequency should we merge ctsm updates into this (these) branches?
      • Every ctsm tag/PR?
      • Only when the new branch is merged into ctsm master?
    • Should PRs be issued to fates-latest? Or ctsm's master?

Latest issues and discussions of note

Calibration progress update

Tutorial update

July 22, 2024

Wins and Celebrations :tada:

  • NGEE-T proposal review complete
  • Land use v2 integrated (E3SM API to be integrated soon)

Upcoming meetings and events :spiral_calendar:

  • FATES modeling meeting: August 15, Summer Office Hours
    • Pick up regular cadence starting after the tutorial in September?

Machine outages :desktop_computer:

  • CISL: August 12-15
  • NERSC: July 31, Perlmutter

PR board status update

New

Updates

Latest issues and discussions of note

Calibration progress update

Tutorial update

  • Beta-testing scheduled for two-weeks starting Aug 12

July 15, 2024

Cancelled due to NGEE-Tropics proposal review

July 8th, 2024

Wins and Celebrations :tada:

Upcoming meetings and events :spiral_calendar:

Machine outages :desktop_computer:

PR board status update

  • No PRs integrated this last week

New

Updates

Latest issues and discussions of note

  • Incorrect per-age-class outputs
    • Consensus is that the _AP should be weighted by the patch age class area, although we are not consistent currently
    • Sam also noting that #1170 looks like it may not have fully addressed the issue for FATES_LAI_AP.

Calibration progress update

  • nothing of note, focus is on preparing for NGEE-T proposal

Tutorial update

  • Greg shifting work post landuse v2 to tutorial infrastructure starting this week

July 1st, 2024

Wins and Celebrations :tada:

  • LU V2 seems to be passing tests, ready for integration soon?

Upcoming meetings and events :spiral_calendar:

Machine outages :desktop_computer:

PR board status update

  • No PRs integrated this last week

New

Reviewers

Updates

Latest issues and discussions of note

No new issues or discussions

Calibration progress update

  • Rosie: Planning to check in with Charlie about handing off efforts to him wrt land use spin up.

Tutorial update

  • FATES tutorial will use FATES API 35

June 24, 2024

Wins and Celebrations :tada:

  • Carbon budget closing thanks to Ryan and Jessie's efforts. PR forthcoming.

Upcoming meetings and events :spiral_calendar:

Machine outages :desktop_computer:

  • CISL: none planned
  • NERSC: June 26, Perlmutter

PR board status update

Integrated

Updates

  • Land use v2: final testing with ctsm and e3sm
  • Understory turnover rate: Approved with one minor comment. Needs testing. Has new parameters.
    • Plan is to have this be its own API update post land use API update.
    • Rosie would like to get this in soon for calibration efforts. As such, we will bring in #1206 and TBD nutrients switch default PR afterwards.
  • Dynamic patch arrays: Approved with one minor question. Needs testing.
  • Grass allometric equations: Check in with Xiulin. Needs testing. Updates param values.
  • Growth Resp to daily timestep: Ask for another reviewer. Needs build and run testing.
    • Jessie will review as well.

Latest issues and discussions of note

No new issues or discussions

Calibration progress update

  • Rosie: Planning to check in with Charlie about handing off efforts to him wrt land use spin up.

Tutorial update

  • FATES tutorial will use FATES API 35

June 17, 2024

Wins and Celebrations :tada:

Upcoming meetings and events :spiral_calendar:

  • FATES modeling meeting: June 20, Summer Office Hours
    • Graduate student intern visiting NCAR this summer will present briefly on pyrogenic carbon
    • Related: Rosie noted that there is a NorESM PR to bring couple trace gasses back to HLM
    • Adrianna to send out email
  • 2024 CESM workshop debrief
    • FATES presentations from Adrianna and Rosie

Machine outages :desktop_computer:

  • CISL: none planned
  • NERSC: June 26, Perlmutter

PR board status update

Integrated: History flush and zero fixes

Updates

Latest issues and discussions of note

Calibration progress update

  • Discussion of determine the uncertainty of the parameters

Tutorial update

Walk on topics

  • Discussion of adapting damage model to fire processes
    • Damage model would likely slow down a global model quite a bit

June 10, 2024

Wins and Celebrations :tada:

Upcoming meetings and events :spiral_calendar:

Machine outages :desktop_computer:

  • CISL: June 11, Derecho
  • NERSC: June 26, Perlmutter

PR board status update

Updates

Latest issues and discussions of note

Carbon balance closure issue

  • Marcos: attempt turning off reproduction to determine if that is the issue.
  • Ryan/Charlie: NPP minus seed decay should equal allocation
  • Ryan: we could also add another data structure to add this up over time

Calibration progress update

no updates

Tutorial update

  • Google group for attendees setup
  • Greg to send out email with "hello world" docker setup request

June 3, 2024

Wins and Celebrations :tada:

Upcoming meetings and events :spiral_calendar:

Machine outages :desktop_computer:

  • CISL: none this week
  • NERSC: June 26, perlmutter

PR board status update

Updates

Latest issues and discussions of note

Issues

Discussions

Calibration progress update

Adrianna: updated emulator based on Jessie's suggestions which showed improvement in estimating average GPP.

Tutorial update

May 27, 2024

Cancelled due to Federal Holiday

May 20, 2024

Wins and Celebrations :tada:

Upcoming meetings and events :spiral_calendar:

Machine outages :desktop_computer:

  • CISL: none this week
  • NERSC: June 26, perlmutter

PR board status update

Integrated: photosytem 2 named constant

Updates

  • Land use v2: addressing some run failures
    • Ryan: B4B expected
  • Growth Resp to daily timestep: review underway
  • Understood turnover rate: prioritization?
    • Rosie would like to see this in for calibration. Jessie has this on her calirbation work currently.
  • Dynamic patch arrays
    • Looking for clarification on the consensus from last week
    • Agreement that ncl of two is too small.
    • Next steps: test performance of ncl of three against dynamic allocation PR

Latest issues and discussions of note

Issues

Discussions

Calibration progress update

  • Rosie: need maintresp, allom_d2ca_coeff, allom_d2bl1 to be low to get LAI to observed
    • It's reasonable for this to be difficult for the model based on literature research
    • Ryan: is radiation model hampering things here? I.e. do we need some sort of explicit gap?
  • Adrianna: shared latest updates from variation on results based on number of gridcells included

Tutorial update

May 13, 2024

Wins and Celebrations :tada:

Jennifer on Star Talk

Upcoming meetings and events :spiral_calendar:

Machine outages :desktop_computer:

  • CISL: none this week
  • NERSC: May 15, perlmutter

PR board status update

Integrated: Allometry unit test, Inventory input updates, e3sm unstructured grid tool

Updates

New

  • Dynamic patch arrays
    • team formulated to discuss and assess impact of setting nclmax
    • termination mortality proposal for cohorts beyond nclmax sounds good
  • Growth Resp to daily timestep: needs build testing
    • TL;DR: If plants aren't growing, they shouldn't be growth respiring. FATES only grows on daily timestep => only growth respiring on daily timestep
    • How does this compare to HLMs?
    • Don't use NEP for carbon budgeting; introduce a NBP flux for carbon budgeting.

Latest issues and discussions of note

  • no new issues or discussions of note

Calibration progress update

  • Adrianna: TRENDY plans discussion
    • TRENDY need: External pft-specific seed rain for extinct pfts (for spin up compensation)
    • Updated SP mode configurations forthcoming

Tutorial update

  • No new updates

May 6, 2024

Wins and Celebrations :tada:

New group using FATES: Tau Liu at Michigan Tech. Teaming up with Chonggang. Interested in calibration and regional runs.

Upcoming meetings and events :spiral_calendar:

Machine outages :desktop_computer:

  • CISL: May 7-8, derecho
  • NERSC: May 15, perlmutter

PR board status update

Integrated: CCH water xfer functions, LAI-by-patchage

Updates

  • Small fire refactor: ready to test
  • Allometry unit testing: testing discussion held on Friday. Minor org refactor to be done.
  • Land use v2: Walkthrough held. Break out land use alignment from disturbance TBD.
    • CTSM API 36: drafted. run mode combo checks discussed and updated
    • How do we test spin up for land use?
      • Similar to the fact that we don't have AD testing
      • Sam Rabin knows how to add a new test in run_sys_test. Greg will tag up with him later.
      • Decision: make issue for testing spin up, but don't hold this up. Add standard regression tests for other landuse v2 modes.
    • Deadline: integrated Friday May 24

New

  • Land Energy Balance Solve
    • what is a good absolute case? Look at distribution of conductance based on different values and come up with reasonable delta.
    • Marcos: Stomata have minimum when close so what about using residual as the absolute tolerance?
    • Ryan: check in with Gautam for E3SM-side reviewer and feedback

Latest issues and discussions of note

  • no new issues or discussions of note

Calibration progress update

Tutorial update

  • No new updates

April 29, 2024

Wins and Celebrations :tada:

  • NGEET Phase 3 proposal submitted
  • First FATES newsletter sent out
  • ELM User's guide PR with connecting FATES documentation

Upcoming meetings and events :spiral_calendar:

Machine outages :desktop_computer:

  • CISL: none planned
  • NERSC: May 15, perlmutter

PR board status update

Latest issues and discussions of note

  • spin-up speed-up
    • Dan Ricutto might now
    • E3SM specific. Loads stuff into memory.
  • growth respiration calc
    • Initial solution consideration (Marcos): calculate on a daily basis and apply with each timestep
      • Need to have code to account for edge cases given if function based on previous day's value
    • We want to get this done prior to the nocomp calibration release
    • How does this apply with HLMs (big leaf)?

Calibration progress update

  • Adrianna: Daniel and Lydia discussion about setting up emulator. How much do we need for a good set for each pft?
  • Rosie: Rerunning one at a time nocomp ensemble with latest main branch (daylength + alt vert scaling).
    • Charlie: High seed rain clarification: Rosie wants to dig into this a bit more, ideally with fully spun up case.
    • Is there a danger in including seed rain in an ensemble for global calibration?
    • Rosie will make global map of seed rain impact

Tutorial update

  • Logistics: Organizing travel

Walk-on topics

  • Should we re-review this meeting time?
    • Goverance meeting?

April 22, 2024

Wins and Celebrations :tada:

Upcoming meetings and events :spiral_calendar:

  • FATES modeling meeting: April 25, Huanyuan Zhang
    • Jessie got confirmation for this week
    • Marcos will need someone to host May 23

Newsletter discussion

  • What sort of topics would people like to see?
    • Add request for fates users
  • How to generate content?
    • Asynchoronous update + in meeting review
    • Dump "wins" from these meetings into the document live
  • Ryan to setup calendar reminder for sending it out every quarter
  • Rosie: Post-doc position - Rosie

Metrics questions

  • How many countries and institutions use FATES?
    • Rosie has a pretty good handle on this. She will send out slide with this information.
      • US universities is an area of uncertainty.
      • Tutorial application could be used for metrics in this regard.
  • How many members do we have?

Machine outages :desktop_computer:

  • CISL: none planned
  • NERSC: May 15, perlmutter

PR board status update

  • API 35 integration and PR1128 is imminant
  • API 33 integration needs confirmation of developer tests passing
  • Landuse v2 walkthrough scheduling
    • Setup walkthrough meeting next Thursday in the timeslot for fates modeling
  • Ryan to reach out to hydro folks for final review
  • Adrianna looking for reviews for unit testing and fire refactor
    • Unit testing needs cime and shr
    • Greg and Charlie to review

Latest issues and discussions of note

  • Minor cleanup issue submitted by Sam Rabin

Calibration progress update

  • Rosie: shared parameter sensitivity results

Tutorial update

  • No new update

Walk-on topics

  • Carbon budget issue - DBEN - Jessie
  • Post-doc position - Rosie

April 15, 2024

Cancelled

April 8, 2024

Wins and Celebrations :tada:

  • Eclipse today! :sun_with_face: :new_moon:
  • Perlmutter speed up trick: DVS in read-only mode (see Jessie's email)
  • Jennifer's birthday

Upcoming meetings and events :spiral_calendar:

Machine outages :desktop_computer:

  • CISL: none planned
  • NERSC: 17 Apr, perlmutter

PR board status update

  • New allometric modes: Greg testing to check b4b issues Ryan seeing. Test reverting code in EDCanopyStructureMod
  • CCH water xfer functions: CCH hydro improvements coming along. Ryan now tracking down issue with non-debug hydro resulting in carbon balance errors.

Latest issues and discussions of note

Calibration progress update

  • Rosie: Adrianna and Rosie discussing approach for next steps using Adrianna's pft files (one-at-a-time no comp simulations)
    • Rosie to integrate changes from Jessie's vert scaling PR as well
    • Workflow discussion
      • Jupyter notebooks are not great from a version control perspective, but nicely steps through ensembles updates

Tutorial update

  • No new update

Hydro / SP mode compatibility (Ryan)

  • Ryan noting that we need to add new features to make hydro and sp mode work together.
    • As such, we shouldn't consider failure in the mode to be a bug, but a feature not yet supported.
    • Greg to take the action to add graceful failure with both modes in API35 and API34 ctsm and e3sm PRs respectively.

Walk-on topics

  • Rosie: Fire BVOC emissions
    • Getting MEGAN to work with FATES: https://github.com/NorESMhub/CTSM/issues/36
      • Keeping fire emission factors in fates parameter file and push calculations in FATES to HLM
      • Rosie currently has a new parameter to map from FATES to MEGAN. Rosie doesn't like that this makes the param file not HLM agnostic.
  • Small conference rooms available in B74 (314 and 202)

April 1st, 2024

Wins and Celebrations :tada:

  • Fire trunk fuel bug fix
  • Allometry parameters are in!
  • Hydro restart bug fix
  • FATES tutorial application notices sent out. We've gotten confirmation back from most of them.

Upcoming meetings and events :spiral_calendar:

Machine outages :desktop_computer:

  • CISL: 2-3 Apr, derecho
  • NERSC: 17 Apr, perlmutter

PR board status update

Latest issues and discussions of note

  • fire trunk issue: fixed per #1180
    • Discussion of impacts to current work.
      • Team is following up with those that were impacted.
    • Adrianna to run coarse global 100 year run to get some assessment of impact. She will update the issue with results and close it out.
    • Future refactor is to not include trunk loading in fuel class
  • SP restart crash: since there is no mort in SP mode, we will avoid these in fix

Calibration progress update

  • Jessie: high biomass issue may be alleviated by SP parameters from Adrianna's work. Also recent fire fix may help as well.
  • Adrianna: Planning on rerunning with new pft composition to address sparse grid limitations

Tutorial update

see wins

Walk-on topics

  • Adrianna: Testing stand structure faster (testing initialization)
  • SSH security heads up: backdoor found inxz-utils version 5.6.0
    • Fedora Linux 41 users should check security alert notice

March 25, 2024

Wins and Celebrations :tada:

Upcoming meetings and events :spiral_calendar:

Machine outages :desktop_computer:

  • CISL: 2-3 Apr, derecho
  • NERSC: 17 Apr, perlmutter

PR board status update

Fire refactor overview

Latest issues and discussions of note

Improving PR integration workflow

  • Brainstorming ideas for reducing PR backlogs
  • Reviews
    • Hold more regular code walk-throughs for large PRs
    • Break-up and stage large PRs if possible
  • Testing
  • Other logistics
    • On-the-fly default parameter file build
    • parameter patch file auto-build/checking

Calibration progress update

deferred

Tutorial update

deferred

March 18, 2024

Cancelled

March 11, 2024

Wins and Celebrations :tada:

Upcoming meetings and events :spiral_calendar:

Machine outages :desktop_computer:

  • CISL: none planned
  • NERSC: Wed 20 Mar, perlmutter
  • OLCF: Tue 19 Mar, center-wide

PR board status update

  • CCH water xfer functions: resolves #1163, but now seeing 1D hydro failure
    • Marcos/Ryan: capping psi_min and th_min discussion?
      • Discussion with Jenny last week (water balance conservation issue).
        • Ryan: Use linear model below zero similar to Van Genukten?
        • More discussion needed
    • Greg to track down regression test failure

Latest issues and discussions of note

Calibration progress update

  • Rosie: discussion of disconnect between tuning and diagnostics packages for land

Fire emission coupling hackathon (Rosie)

  • Goal: get fire emissions out of fates and into atmosphere model
  • Calculate biomass burned and pass to HLM to calculate emissions or fates calculate emissions directly
    • emissions are pft specific suggesting we should have fates calculate directly
      • would result in duplication of emissions factors from HLM
  • Charlie: is emission a function of fire "type" (i.e. anthro versus wildfire)
    • Rosie: yes, but pft is a proxy of this
  • Ryan: discussion or issue?

Hydaulic failure discussion (Chonggang)

  • Option: when leaf water potential high, zero stomatal conductance and not request water
    • May improve numerical stability
    • Chonggang hoping to attempt this at the end of the month
  • Ryan: preference is to implement linear model between
    • Alternate (short term?): run case at much shorter time steps to get through snags
  • Charlie: Develop dry condition test failures (e.g. single site at Atacama)
    • Ryan: functional unit testing
      • satphen + hydro?
    • Greg: being able to parameterize functional unit tests would be great

Tutorial update

  • deferred

March 4, 2024

Wins and Celebrations :tada:

Upcoming meetings and events :spiral_calendar:

Machine outages :desktop_computer:

  • CISL: Tue 5 Mar, derecho
  • NERSC: Wed 20 Mar, perlmutter
  • OLCF: Tue 19 Mar, center-wide

PR board status update

  • Stuck:
  • Update:
  • Discussion:
    • Vertical scaling: Discuss email thread on clarifying scalar definitions and migration to parameter file
    • Action:
      • Ryan to create new issue for discussion about sla scaling in the biophyscial rates routine
      • Jessie: add parameters Ryan suggested to PR to tee up for future implementation without needing to worry about future API update.

Latest issues and discussions of note

  • No new issues or updates

Calibration progress update

  • No new issues or updates

Global stomatal conductances (Ryan)

  • Making coupling HLM-FATES faster. Honing in on the right stomatal conductance.
    • Noticing how wide range of values are globally. Why values in the sahara (full fates)?
    • Ryan will run an sp mode and share results

Tutorial update

Walk-on topics

  • Rosie: mini hackathon next Thursday/Friday to tackle the fire emissions coupling.
    • Add to discussion agenda for next Monday.

February 26, 2024

Upcoming meetings and events

FATES software meeting check-in

  • Is there interest in moving this meeting to an every-other week cadence?
    • Consensus: those present like the weekly cadence and the reporting back of PRs and issue status.
    • Greg: What about the notes format? Would people prefer to engage with the notes in another format?
      • Consensus: those present like the transparency of the wiki format and that the notes are co-located on the repo with everything else (issues, discussion, code, etc.)

Machine outages

  • CISL: none planned
  • NERSC: 20 Mar, perlmutter
  • OLCF: none planned

PR board status update

Updates:

Latest issues and discussions of note

  • #1163 and #1165 addressed by PRs #1164 and #1163, respectively

Calibration progress update

  • Jessie: finishing up respiration work; will turn into maniscript in near future.
    • Considering adoption of Adrianna's ML method for next iteration (post SP-mode calibration updates)
    • Ryan: stem respiration for calibration?
      • Jessie: Not currently. Kolby keen to investigate this for next project phase. Discussions about field-to-model integration details on going.

Tutorial update

  • Jennifer: what's the accessibility goal? Will we explain how to setup the container, etc?
    • Jessie: Ultimate goal is to make this available all online with a very simple BCI method.
      • Greg: We also have a tutorial setup guide repo in development to host a readthedocs style walkthrough of setting up on a laptop.
    • Greg: OLMT experience? We might use or compare the single site subsetting function for pre-processing.
      • Jennifer: Yes, two years ago. Current attempts to use were hampered however.
      • Ryan: Running pre-process all at once?
        • Jessie: yes, we could use the sparse grid NCL method.

February 12, 2024

Upcoming meetings and events

Machine outages

  • CISL: none planned
  • NERSC: 21 Feb, perlmutter
  • OLCF: none planned

PR board status update

Latest issues and discussions of note

FATES speed audit update (Ryan)

  • Can we avoid leaf layer loop: Trimming routine forces us to iterate over leaf layer depth
  • Current plan: investigate Ci iteration cap and threshold. Update as necessary along with remembering the previous value.
  • Initial iteration analysis update suggests that we often hit the iteration cap.
    • Initial iteration testing with ST3 mode (audit was using full dynamics however).
  • ED2 experience: What about trying Newton method using the derivitives from reduced analytical form? Review the ED2 approach and share with team.
  • Investigate impact of making photosynthesis loosely coupled?

Calibration progress update

  • Adrianna: CLM surface data does not seem to quite match ILAMB observation. This is causing issues with gridcell to gridcell comparison.
    • Ask Nate Collier for help.
  • Calibration team to review Adrianna's brute-force method and will review selected parameter ranges.

February 5, 2024

Upcoming meetings and events

Machine outages

  • CISL: 5-7 Feb, derecho
  • NERSC: 21 Feb, perlmutter
  • OLCF: 6 Feb, frontier

PR board status update

Latest issues and discussions of note

Calibration progress update

January 29, 2024

Upcoming meetings and events

Machine outages

  • CISL: Mon 5 - Wed 7 Feb, derecho
  • NERSC: none planned
  • OLCF: none planned

PR board status update

Latest issues and discussions of note

  • LU data tool fails on derecho: probably taking too much memory for login node. This will need to be addressed eventually for CLM LU build on the fly.
    • This came up in the context of checking ctsm python environment on derecho, due to a dependency issue for xesmf.
    • Erik/Charlie: test this on the develop queue

Calibration progress update

  • Adrianna: Ran Ball-Berry without temp acclimation. Observational mean is more easily achieved, but emulator does a worse job.
    • Temperture acclimation without very low vcmax
      • Charlie: Claire Zarackas is finding something similar in the absence of hydraulics. So there is a scientific rationale for excluding temp acclimation.
  • Jessie: met with Ryan to talk about carbon budget issue; add recruitment carbon flux. Carbon budget closes now.
    • Not sure if recruitments that die in the same timestep are showing up in the diagnostics

Walk-on topics

  • Rosie: Question: Matvey fates model focus? Take over Marius work? Where does it fall in the priority?
    • Jennifer: arctic-fire stuff coming up prior to arctic-hydro. Maybe hardening?
    • Greg: This is back on API 17, so bringing it up before it gets too stale would be good
    • Charlie/Ryan/Rosie: recent hydro issues do suggest that addressing hardening is a good idea.
    • Matvey: only calibrated to 2 local sites; will it work globally?

January 22, 2024

Upcoming meetings and events

Machine outages

  • CISL: Mon 5 - Wed 7 Feb, derecho
  • NERSC: none planned
  • OLCF: none planned

PR board status update

Building parameter files on the fly

  • Ryan to look at moving this forward this week

Latest issues and discussions of note

  • hydro dry region failure: Marcos has been working with Jenny on this. Recommendation is that she not trace the issue since its out of scope for her work.
    • Ryan proposing to add a fail safe for extreme outlier cases to allow fates run to complete. Still need to make sure this is actually an outlier.

Calibration progress update

  • Calibration branch
    • Jessie still needs this branch until 2-stream gets to E3SM (probably a few weeks at this point). The calibration branch is pretty close to FATES main for her purposes, so no need to merge main into calibration branch.

Walk-on topics

  • Erik asking about status of Marius' frost branch

January 15, 2024

No meeting due to lab holiday

January 8, 2024

Upcoming meetings and events

Machine outages

  • CISL: Mon 5 - Wed 7 Feb, derecho
  • NERSC: Wed 17 Jan, perlmutter
  • OLCF: none planned

PR board status update

Latest issues and discussions of note

Calibration progress update

  • Jessie presented latest results: GPP and biomass bias improvements

January 1, 2024

No meeting due to lab holiday