2024 Software Engineering meeting notes - NGEET/fates GitHub Wiki
December 16, 2024
- Wins (1 min)
- Machine status and meeting notices (5 min)
- PR board status (10 min)
- Latest issues (5 min)
- Calibration update (10 min)
Wins and Whimsy :tada:
- Former tutorial student requested to use fates tutorial for Ghent University vegetation moduleing course
- Jessie and Greg to reply. Add Ryan and Adrianna to the reply as they are interested in the course content as well.
Machine status and notices :desktop_computer:
Upcoming meetings and events :spiral_calendar:
- FATES modeling meeting: Next meeting January 16?
- Consensus: yes
- Jessie to reschedule with Anne-Marie
PR board status update
- Greg and Ryan to schedule longer outlook planning session for PR board in the new year
Integrated
Update
- E3SM API37 (E3SM#6762): Gautam confirmed to be integrated this week.
- Refactor rate of spread: testing is b4b, but needs review
New
- parameter switch migration
- To be coordinated with ESCOMP/CTSM#2904
- CTSM PRs waiting on high priority PR ahead of this
- E3SM-side pull request TBD
- CK: Should we update defaults at the same time? E.g. Medlyn.
- RK: Jenny has update to default.
- GL: We can, although there is an opportunity to bring this in on the ctsm
b4b-dev
branch if we don't- That said, doing so is dependent on some other ctsm queue factors
- CK: how should we handle real number parameters that are switch dependent? How to make the user aware?
- Differences between defaults for the host land models will eventually exist in parameter file
- RK: This is a short-term solution. Should we accelerate moving to migrating parameter file to xml?
- This would allow us to nest and check parameters based on the switch
- Consensus is moves this forward as it solves current testing issues. Conduct
- To be coordinated with ESCOMP/CTSM#2904
- Don't zero
patch%fuel%frac_burnt
: needs review- Fixes issue #1301
- CK: Fuel amount conditional on patch age. Adrianna to review.
- Fix cmake for functional tests
- AF: note that if you are running with latest ctsm tag, functional tests will break without this PR
issues and discussions of note
Latest- inventory restart question (fates-tutorial#64): feedback to our questions recieved, to be reviewed
- FDI calculation: discuss next time
- fire calculations regardless of event: discuss next time
Calibration progress update
- deferred for walk on topic
Walk-on topic
- Rosie: previewed FATES diagnostic work
- TVEG issue discussion: sp mode looks ok, nocomp is too low.
- RF: running with default parameters
- RK:
hlm_hio_ignore_val
should not be averaged in - CK: make a map of equivalent host land model to see how it compares. Should align with fates.
- TVEG issue discussion: sp mode looks ok, nocomp is too low.
December 09, 2024
Cancelled due to AGU
December 02, 2024
- Wins (1 min)
- Machine status and meeting notices (5 min)
- PR board status (10 min)
- Latest issues (5 min)
- Patch Allocations and Boundary Arrays in Satphen, Fixed Biogeo and nocomp
- Calibration update (10 min)
Wins and Whimsy :tada:
Machine status and notices :desktop_computer:
Upcoming meetings and events :spiral_calendar:
- FATES modeling meeting: AGU Lightning Talks, 5 December
- Charlie to to solicit presentations
- AGU Annual Meeting: 09-13 December
PR board status update
Integrated
Update
- radiation singularity fix: To be integrated.
- E3SM API37 (E3SM#6762): To be integrated.
- leaf biophys refactor (#1262): Adrianna confirms she isn't seeing any new crashes.
issues and discussions of note
Latest- inventory restart question (fates-tutorial#64)
- Greg and Ryan to ask clarifying questions
- Suggest toggling the hard-coded switch to get more diagnostics
- SP mode + finidat case failure (Adrianna)
- Height coming from hlm is zero for some reason in sp mode when running a case copied from a previous case
- Restarting from original case does not have this problem
- Adrianna to create an issue (if necessary)
- Height coming from hlm is zero for some reason in sp mode when running a case copied from a previous case
Calibration progress update
- Adrianna to send python code to Jessie for sparse grid mapping
Patch allocation and boundary arrays discussion (Ryan)
Walk-on topic
November 25, 2024
- Wins (1 min)
- Machine status and meeting notices (5 min)
- PR board status (10 min)
- Latest issues (5 min)
- Calibration update (10 min)
Wins and Whimsy :tada:
Machine status and notices :desktop_computer:
Upcoming meetings and events :spiral_calendar:
- FATES modeling meeting: AGU Lightning Talks, 5 December
- AGU Annual Meeting: 09-13 December
PR board status update
Update
- CNP allocation: regression testing done. Greg may add a unit test.
- radiation singularity fix: Ryan reviewing results
- leaf biophys refactor (#1262): Discussion about capping
New
- #1289 bug fix: Shijie submitted fix
issues and discussions of note
Latest- total fuel incorrect (#1288)
- Logging direct mort condition check bug (#1289)
- Consensus: create a temporary variable per Adrianna's suggestion
Calibration progress update
- Adrianna: Discussion of recent results
Walk-on topic
- Rosie: stomatal conductance and hydro model discussion
- Adrianna is point of contact for upcoming fates work for Danica's grad student
- Invite Tom Buckley to fates modeling meeting: https://onlinelibrary.wiley.com/doi/10.1111/pce.15030
November 18, 2024
- Wins (1 min)
- Machine status and meeting notices (5 min)
- Activity planning board feedback (10 min)
- PR board status (10 min)
- Latest issues (5 min)
- Calibration update (10 min)
Wins and Whimsy :tada:
- Adrianna's first FATES merge and tag.
- NorESM merged MEGAN-FATES update to their ctsm branch. Will be merged back to ESCOMP ctsm.
Machine status and notices :desktop_computer:
Upcoming meetings and events :spiral_calendar:
- FATES modeling meeting: Ensheng Weng, 21 November
- AGU Annual Meeting: 09-13 December
Activity planning board
- Initial thoughts
- Advertise board at modelling meeting and in newletter
- Inviting people to contribute?
- Greg to generate google form for people to have their project added
- Archiving and converting items to issues? Consensus: yes
PR board status update
Integrated
- Spitfire fuel calc refactor (#1247), Growth Resp to daily timestep (#1197), calculate sapwood cross-sectional area (#1268)
Updates
- land use fixes (#1273): Final review this week. Next in queue for testing.
- leaf biophys refactor (#1262): ready for review
- Refactor per-ageclass (#1252): Ryan double checked units for new
SumMortForHistory
function
New
- FatesPFTIndexSwapper.py fix (#1287)
- Remove #1254 workaround in hydro tests (CTSM#2882): quick testing fix and updates ctsm to point at tag for #1268
- 2-stream finidat fix (#1286): Ryan to redo with cherry-picked commits (avoid Grazing commits)
issues and discussions of note
Latest- lighnting IO performance (#1285): Fates-side tracking issue to E3SM#6742
- 100% shrub reproduction crash (#1283): Discussion about potential fix approaches. Jessie to open a PR to get conversation started.
- Charlie: had a break through on land use and nutrients issue.
Calibration progress update
- Rosie: making updates to ensemble
Walk-on topic
- Dynamic column/land unit areas: discuss Ryan's proposal versus original design idea
November 11, 2024
- Wins (1 min)
- Machine status and meeting notices (5 min)
- Need for a more forward-facing project board
- PR board status (10 min)
- Btran controls Slide 11
- Latest issues (5 min)
- Calibration update (10 min)
Wins and Whimsy :tada:
Machine status and notices :desktop_computer:
Upcoming meetings and events :spiral_calendar:
- FATES modeling meeting: Ensheng Weng, 21 November
- AGU Annual Meeting: 09-13 December
Forward looking project board (Charlie)
- This came out of a discussion between Charlie and Adrianna.
- Both forward-looking (planning) and forward-facing (outreach)
- A lot of our boards are for current efforts (e.g. PR board, Landuse board)
- We want community to know what we are doing and know how to reach us for collabs
- Jessie suggested reviewing this at the modeling meeting to encourage used
- Allowing people to add their own items
- Greg to sketch out the board which we'll review in this meeting next Monday
PR board status update
Updates
- Spitfire fuel calc refactor (#1247): investigating segfault issue.
- Refactor per-ageclass (#1252): Greg reviewed with minor comments/questions. To be tested.
- fates hydro solver param update (#1263): Global 100-year check looks good.
- Ryan had some clarification on the results and her recommendation.
- calculate sapwood cross-sectional area (#1268): Greg or Ryan to also review. Ping Zachary post review.
- Growth Resp to daily timestep (#1197): Greg to test this week.
- Grazing: Charlie running sensitivity analysis. Needs review.
- Question to Sam R to calibrate. Sam used one of human appropriation PPE papers.
- Leaf Biophys Unit Tests: BTRAN controls discussion.
- Rosie: Sabot et al paper for comparison
- Ryan to add alternative hypothesis to the PR.
- Also: Adrianna's test run resulted in failure. Ryan to address via bisection for c4 grasses.
New
- HLM API table update (fates-users-guide#79): reorganization of table
- finidat with cold start fix (CTSM#2870): Sam working on this
issues and discussions of note
Latest- 100% shrub reproduction crash (#1283): allocation issue. Ryan to address.
- History clean up super task (#1282): use this super task wishlist to collect small history clean up tasks
- History order of operations refactor (#1281): Ryan and Greg discussed refactoring history order of operations. This super task to capture subtasks.
- Sam R notes this aligns with his idea of having conceptually aligned history writes.
- Charlie's Restart Error: Need PRs created from the branches in the last comment
- As a reminder, the finidat issue here has to do with TwoStream. Different from CTSM#2870.
Calibration progress update
- No updates
November 04, 2024
- Wins (1 min)
- Machine status and meeting notices (5 min)
- PR board status (10 min)
- Latest issues (5 min)
- Calibration update (10 min)
Wins and Whimsy :tada:
Machine status and notices :desktop_computer:
Upcoming meetings and events :spiral_calendar:
- FATES modeling meeting: Annemarie Eckes-Shephard, 7 November
- CTSM-FATES priority board discussion: 7 November 9:30 AM PT (CSTM SE meeting)
- AGU Annual Meeting: 09-13 December
PR board status update
Integrated
- Refactor unit testing scripts (#1245), Inventory init write fix (#1244), Old land use tool clean up,
Updates
- Spitfire fuel calc refactor (#1247): Testing underway
- Growth Resp to daily timestep (#1197): Needs deconflict and testing
- Consensus check in on moving forward with this integration process. Agreed to move forward.
- fates hydro solver param update (#1263): Yiling reviewed and approved. Jenny to run global low-res check.
- calculate sapwood cross-sectional area (#1268): Zachary to review
- Leaf biophysics discussion
- Criteria is too stringent, needs to be relaxed. Number of iterations should be increased.
- Solution is smoother now
- Need to identify a reasonable convergence tolerance.
- Ryan to run new branch in Medlyn smoke test GSWP3, 2000 to try and replicate error Adrianna is seeing with Medlyn out of the box
- Charlie: need a PR to update nclmax
issues and discussions of note
Latest- Adrianna: imaginary quadratic solve issue
- Rosie: finidat issue
- Rosie: testing + compset concerns from Marianna
Calibration progress update
Walk-on topics
October 28, 2024
- Wins (1 min)
- Machine status and meeting notices (5 min)
- PR board status (10 min)
- Latest issues (5 min)
- Calibration update (10 min)
Wins and Celebrations :tada:
Machine status and notices :desktop_computer:
Upcoming meetings and events :spiral_calendar:
- FATES modeling meeting: Annemarie Eckes-Shephard, 7 November
- AGU Annual Meeting: December 09-13
PR board status update
Integrated
Updates
- Refactor unit testing scripts (#1245): Tested, to be integrated
- Spitfire fuel calc refactor (#1247): Adrianna to update per review comments
- Inventory init write fix (#1244): Ryan approved, Greg to test
- Refactor per-ageclass (#1252): Sam to address Ryan's comments and schedule a meeting to discuss
TODO
s
New
- fates hydro solver param update (#1263): Needs reviewer. Ryan to reach out to hydro folks.
- calculate sapwood cross-sectional area (#1268): Fixes #1254, needs reviewer
- Large LAI crash fix (#1269): Needs reviewer
- Add cohort method to encapsulate check and pull out of sai itself
issues and discussions of note
Latest- sapflow bug (#1266): Discrepency between
FATES_SAPFLOW_SZPF
andFATES_SAPFLOW
- Charlie/Marcos: Units are incorrect
- Branch runs failure (#1271): Sam tested and found that branch runs fail for FATES. Tested using
ERI
prefix, for which no fates test exists.
Calibration progress update
- Adrianna: GPP improvement globally (GBAF, FLUXCOM, WECAN). Slightly under calculating in the amazon now. Much better everywhere else including Africa.
Walk-on topics
October 21, 2024
- Wins (1 min)
- Machine status and meeting notices (5 min)
- PR board status (10 min)
- Low humidity discontinuity discussion (10 min)
- Latest issues (5 min)
- Calibration update (10 min)
Wins and Celebrations :tada:
Machine status and notices :desktop_computer:
Upcoming meetings and events :spiral_calendar:
- FATES modeling meeting: 24 October, Luke Oberhagemann
- Non-steady state stomatal conductance paper: Marcos to reach out and coordinate
- AGU Annual Meeting: December 09-13
PR board status update
Integrated
Updates
- E3SM API36.1 (E3SM#6639): Approved, tested. Waiting integration.
- new patch insertion method (#1155): Approved and tested. Diffs need review.
- Ryan: we need some sort of order protection so we don't make assumptions about age ordering
- Growth Resp to daily timestep (#1197): Needs deconflict and triage of Jessie's suggestions.
- Excess respiration should be merged into this.
- Refactor unit testing scripts (#1245): Adrianna requested moving this before #1247 fire refactor. Greg to review.
- Inventory init write fix (#1244): Ryan to review
- Refactor per-ageclass (#1252): Out of WIP and ready for review
- All history PR (CTSM#2601): Check in with Sam
- Future enhancement (Ryan): Update
fates_history_dimlevel
to accept negative values to include inactive variables for given dimlevel? - Sam agrees that this would be good. Greg to make an issue to track the enhancement.
- Future enhancement (Ryan): Update
Low relative humidity discontinuity discussion
- Discovered via Demo of Leaf Biophys Unit Tests
- Are the discontinuities due to singularities in stomatal conductance calculation?
- Marcos has seen this in ED2 as well and addressed it.
- Dynamically make sure that the initial guesses don't straddle the singularity.
- Only an issue with Ball-Berry. Move to medlyn as the default?
- Adrianna: Melyn has a correction that doesn't allow humidity to get that low (cap on VPDs between leaf and atm, not atm itself).
- CLM apparently fixed the issue and removed the cap.
- Ryan removed the caps for medyln and didn't see the issue reappear.
- That said Adrianna is seeing convergence errors.
- Has already been documented: https://github.com/NGEET/fates/issues/1118.
- Bisection algo from Gordon might resolve some of these per Ryan?
- CLM apparently fixed the issue and removed the cap.
- Ryan and Marcos to setup a meeting to discuss
- Adrianna: Melyn has a correction that doesn't allow humidity to get that low (cap on VPDs between leaf and atm, not atm itself).
issues and discussions of note
Latest- Grass sapwood allometry hydro issue (#1254): Xiulin to create PR to address this soon.
- Biomass outputs updating for non-new cohort only
- Ryan: if its a state should be included, but if its a flux it shouldn't be used. We need to do an audit of Sam's list and what is/isn't a flux.
- Marcos: Issue is more if its per plant rather than per area.
- Future refactor: splitting history interface mode between fluxes and states
- Immediate work: audit the list and create PR to correct. This should be based on Sam's PR and come in afterwards.
- Ryan: if its a state should be included, but if its a flux it shouldn't be used. We need to do an audit of Sam's list and what is/isn't a flux.
- FATES simluation in desert environs: Rosie answered. Cross-post low relative humidity issue here as well.
Calibration progress update
- Adrianna: getting results for co-dominant
Walk-on topics
- Sam: distinction between subroutine hist dyn 1 and 2?
- Adrianna: suggests renaming the subroutines to be more descriptive
October 14, 2024
- Wins (1 min)
- Machine status and meeting notices (5 min)
- PR board status (10 min)
- Latest issues (10 min)
- Demo of Leaf Biophys Unit Tests
- Calibration update (10 min)
Wins and Celebrations :tada:
Machine status and notices :desktop_computer:
Upcoming meetings and events :spiral_calendar:
- FATES modeling meeting: 24 October, Luke Oberhagemann
- AGU Annual Meeting: December 09-13
PR board status update
Integrated
Updates
issues and discussions of note
LatestCalibration progress update
Walk-on topics
October 07, 2024
- Wins (1 min)
- Machine status and meeting notices (5 min)
- PR board status (10 min)
- Latest issues (10 min)
- Calibration update (10 min)
Wins and Celebrations :tada:
- Marcos recieved the Thomas Hilker Early Career Award
- Ryan released the second newletter!
Machine status and notices :desktop_computer:
Upcoming meetings and events :spiral_calendar:
- FATES modeling meeting: October 10, Steven de Hertog
- AGU Annual Meeting: December 09-13
PR board status update
Integrated
- Arctic shrub PFT additions (#1236), Understory turnover rate (#1136)
- merge PR (#1255): Used a merge PR to coordinate, update, and automerge the two PRs.
- CTSM API 36.1 update (ESCOMP/CTSM#2700)
Updates
- Time integrated flux diag (#1217): Ryan to address review comments.
- E3SM API36.1 (E3SM#6639): needs review, testing
- new patch insertion method (#1155): Greg to review and more forward
- Growth Resp to daily timestep (#1197): Ryan to incorporate respiration excess
- Refactor per-ageclass (#1252): Sam targeting work on this
issues and discussions of note
LatestGrass sapwood allometry hydro issue (#1254)
Calibration progress update
- Charlie: Seed rain impact on spin up discussion (wrt nocomp calibration)
- Seeds may be supporting a population of plants that are constantly dying
- Should we be making some metric/ecosystem benchmarks to make sure that certain pfts are being represented
- It appears that die-off is mostly in grasses
- Investigate using Xiulin's new grass allometry
- Charlie to update with more detailed maps showing the differences
- Maybe test seed dispersal
Walk-on topics
- Rosie: FATES reading 78 pft issue
- NorESM-FATES needs 16pft file for spectral element simulations as well
- Discussion triggered by issue (#2810) testing showing that we needed a file for ctsm5.3.0.
- Sam L and Mariana have created to address.
September 30, 2024
- Wins, Wins, more Wins, a few more Wins, and then another Win (lets just end there?)
- Machine status and meeting notices (5 min)
- PR board status (10 min)
- Latest issues (10 min)
- Calibration progress update (10 min)
Upcoming meetings and events :spiral_calendar:
PR board status update
Updates
issues and discussions of note
LatestCalibration progress update
Walk-on topics
September 23, 2024
- Wins (1 min)
- Machine status and meeting notices (5 min)
- PR board status (10 min)
- Latest issues (10 min)
- Newsletter
- Calibration progress update (10 min)
Machine status and notices :desktop_computer:
Upcoming meetings and events :spiral_calendar:
- FATES modeling meeting: September 26, Bianca Rius - CAETE model
PR board status update
Updates
- Arctic shrub PFT additions (#1236): queueing and regression testing
- Jennifer: running an additional smoke test to conduct some difference mapping
- updates are a result of observation data and FASST optimatization
- Ryan: coordinating with Adrianna's sp mode update
- Adrianna current work doesn't include bl ever extra trop, and broad decid extra trop
- Since so much is changing with sp mode calibration inputs (e.g. gswp3 -> cru), might as well update this and redo sp calibration
- Adrianna current work doesn't include bl ever extra trop, and broad decid extra trop
- Jennifer: Drive to get this in is to meet NGEE-A milestone
- Greg: start regression testing to get ahead of any other issues since API updates are coming up soon in queue
- Greg to start e3sm API update
- Time integrated flux diag (#1217): under review
- Jessie to add review later today
- new patch insertion method (#1155): under review, Ryan may implement recommended changes for him
- Spitfire fuel calc refactor (#1247)
- Adrianna: question about historical context for zero fuel code comment
- Adrianna will update this to allow for zero fuel during her refactor
- Adrianna: question about historical context for zero fuel code comment
issues and discussions of note
LatestNo new issues or discussions of note
Newsletter feedback
- Marcos: should we be advertizing FATES presenations as well?
- Ryan to add call for people to provide highlights for a presentation and papers
- Marcos to create google form to provide consistent input results
- Goal: get your edits in by Friday. Ryan will send out then.
Calibration progress update
- Jessie, Rosie, and Adrianna to meet on Wednesday
- Linea and Adrianna to create repo for calibration
Walk-on topics
September 16, 2024
- Wins (1 min)
- Machine status and meeting notices (5 min)
- Tutorial retrospective (10 min)
- PR board status (10 min)
- Latest issues (10 min)
- Calibration progress update (10 min)
Wins and Celebrations :tada:
- 2024 FATES Tutorial event was a success!
Machine status and notices :desktop_computer:
Upcoming meetings and events :spiral_calendar:
- FATES calibration coordination meeting: CLM meeting, September 19
- FATES modeling meeting: September 26, TBD
- Ask Steven, who will be here at LBL, if he'd like to present in October
- Bianca Rius for a model comparison. Marcos to reach out.
- Reach out to Yanyan when Rosie is back
Tutorial Retrospective
- Three attendees have signed up on the FATES model google group so far
- Generated a couple issues and started up some PRs during the tutorial
Initial feedback
- Just under half responded via survey so far: mostly positive feedback, some constructive critism
- Software: Docker setup pre-tutorial could use improvement; more introductory material on working in the terminal
- Presentations: Slow down speech cadence during presentations and spend a little more time during intro talks
- Balance of time between hands-on practice and presentations: slight bias towards more presentations with examples of use
FATES team feedback
- Ryan: Presentations as breaking up the day
- Jessie: more presentations on different processes in fates (guided examples)
- Ryan: make it more clear that they can take breaks when they want
- Sam (via Jessie): have a section on debugging. E.g. where are the log files?
PR board status update
Updates
- Arctic shrub PFT additions (#1236): testing updates
- CTSM API 36.1 update (ESCOMP/ctsm#2700): incorporate arctic shrub?
- Peter Thorton keen to get this into E3SM sooner rather than later
- Also wants documentation on this; Jennifer to handle.
New
- Inventory init write fix (#1244): discovered during tutorial
- Refactor unit testing scripts (#1245)
- User's guide updates: necessary updates identified during the tutorial
- TBA: Steven's ED -> FATES inventory conversion script from the tutorial
- Marcos to work with Steven to bring this in
issues and discussions of note
Latest- NorESM-FATES in PEcAn (PecanProject/pecan#3372)
- NorESM land sites (container) platform: https://noresmhub.github.io/noresm-land-sites-platform/
- TRS mass balance error (#1246)
- Negative leaf litter (#1249)
- Veg burial by snow (#1250)
- Adrianna helped Katya out with some initial troubleshooting. Two pronged issue. Confirmed that forcing data is identical.
- Marcos: maybe canopy intersection differences might be worth looking into
- Ryan: head capacity in FATES issue?
- Grazing fix coming soon. Ryan to make PR for fixing #1234.
Calibration progress update
- Adrianna: no major updates due to machine outage. Coordination meeting coming up this week.
Walk-on topics
- Sam: refactoring history PR work
- Having trouble reproducing result from the baseline. Needs a second pair of eyes to review.
- Ryan to tag up with Sam.
- Having trouble reproducing result from the baseline. Needs a second pair of eyes to review.
September 9, 2024
Cancelled due to FATES Tutorial event
September 2, 2024
Cancelled due to Federal Holiday
August 26, 2024
- Wins (1 min)
- Machine status and notices (1 min)
- PR board status (5-10 min)
- Latest issues (5 min)
- Calibration progress update (5-10 min)
- Tutorial update (5 min)
Wins and Celebrations :tada:
Machine status and notices :desktop_computer:
Upcoming meetings and events :spiral_calendar:
- FATES modeling meeting: September 26, return to regular cadence
- Charlie added Yanyan Cheng recent paper. Ask Rosie about having them present paper.
- Jessie schedule Anna Marie in November. Waiting for confirmation.
- FATES calibration coordination meeting: CLM meeting, September 19
- FATES software meeting: move to every other week cadence?
- Informal poll: No strong feelings. Meeting is well run and meeting weekly works well.
- Check in with Rosie and others about finding a better time for the future.
- Informal poll: No strong feelings. Meeting is well run and meeting weekly works well.
PR board status update
Updates
- Fire Weather Refactor: Updates made and retested. Results as expected.
- To be integrated next. No need to wait for CTSM API updates.
- Arctic shrub PFT additions: Should this update the default or have a new cdl file? If not, how do we test this?
- Making default cons: not tested pan-arcticly (one arctic site only), would complicated calibration efforts (?)
- Adrianna: there have been multiple parameter file updates during calibration, so not that big of a concern for SP calibration work
- Check in with Rosie about non-comp impact
- Multiple parameter files? Consensus is no; either update default or just provide xml for this PR.
- Testing with xml only option: we could test this using the parameter auto-gen workflow
- TODO: run long global 4x5 smoke test. If results look good, make this default.
- Making default cons: not tested pan-arcticly (one arctic site only), would complicated calibration efforts (?)
- CTSM API 36.1 update: Waiting on higher priority ctsm-side PRs to meet CTSM5.3.0 tag
New
- Old land use tool clean up: remove out-dated land use tool and add new tool as submodule
- Land use tool clean up: Sam R provided cleanup PR. Testing on 5x5 grid is b4b.
issues and discussions of note
Latest- CanopyStructure third canopy layer
- Added new clause is causing issues due to tolerance. Some version of check should be performed, but not sure what tolerance.
- Ryan: imperfect plasticity would help this, but radiation scattering makes that difficult.
- Charlie: leave open; needs more investigation.
- Added new clause is causing issues due to tolerance. Some version of check should be performed, but not sure what tolerance.
- EDPFTVarcon merging: Ryan will address this.
- FATES inventory pft error: Greg to reach out to author to ask for files
- Charlie's Restart Error: under investigation; Ryan reproduced error
- Ryan: Not all of the params the drive 2-stream have reasonable values on the restart. Adding error codes to exit back into calling subroutine.
Calibration progress update
Adrianna: Shared recent results on broadleaf evergreen to get feedback
- Atkin and stomatal conductance highly correlated.
- Jessie: is this related to odd behavior between Atkin and stomatal conductance that Polly emailed about?
- Workflow only took an hour
Tutorial update
- Beta-testing underway
- Windows testing nearly complete; no show stoppers
- Containers not working on macbook silicon chips
- Need to rebuild baseos with multi-platform support
- FatesPFTIndexSwapper issue came up in testing; likely a dependency version issue
Walk-on topics
August 19, 2024
- Wins (1 min)
- Machine status and notices (1 min)
- Report in on NGEE Arctic, ESM, and ATPESC meetings (10 min)
- PR board status (5-10 min)
- Latest issues (5 min)
- Calibration progress update (5-10 min)
- Tutorial update (5 min)
Wins and Celebrations :tada:
Machine status and notices :desktop_computer:
Upcoming meetings and events :spiral_calendar:
- FATES modeling meeting: September 26, return to regular cadence
- Stephenn deHertog to be here for a month after the tutorial. Marcos to invite him to talk on this date.
- FATES calibration coordination meeting: CLM meeting, September 5
- Goal: coordinate existing effort and timelines
PR board status update
Integrated
Updates
- Turnover rate: testing complete
- CTSM API 36.1 update: testing underway with #1136 and #1206 param update
- Fire Weather Refactor: review underway
- refactor per-age-class outputs: to be closed and reworked post meeting with Ryan and Charlie
New
Future
- Chonggang: Beetle code forthcoming
- Chonggang: Water potential based on salinity just getting started
- Rosie: Has done experimental water potential work with mangroves
- Sam: Prioritizing escaped fire work
- Sam to work on replicating Rosie's fire work (3 pfts)
issues and discussions of note
Latest- 2-stream restart error: testing on
main
with Charlie's minimum script is passing- Charlie: Issue has to do with grazing. He's confirmed that removing grazing alleviates the issue he was seeing.
- Charlie: canopy structure issue (to be )
- There is a check missing that needs added to make sure that the canopy is full before adding to lower canopy layers
- Immediate fix is to change order of promotion from lower to upper
- Whats the reasonable tolerance level on check?
- Rosie: check the largest error for crash
- Prioritization of conservation values (dbh, mass, etc) resulting in conflicts
- Marcos: ED2 doesn't conserve DBH and back-calculates. FATES is more complicated.
Calibration progress update
- Adrianna: No major updates. Script for dominant and co-dominant grid working.
- Greg: documentation of process and tools used? Maybe this should be a discussion point for the coordination meeting?
Tutorial update
- Beta-testing started. Greg conducting OS testing on Windows. Jessie receiving inventory coordinates to generate surface/domain data subsets.
Walk-on topics
- Rosie: AD spin-up use recently (in CLM-FATES)? Seeing crashes.
- Charlie: should work, although had to comment out some phosphorous code in ELM for something specific
- Adrianna suggested looking in CESM forum. She recalls seeing some issue posted there.
- Chonggang: Topounits with FATES?
- They should theoretically work together
August 12, 2024
- Wins (1 min)
- Machine status and notices (1 min)
- Report in on NGEE Arctic, ESM, and ATPESC meetings (10 min)
- PR board status (5-10 min)
- Latest issues (5 min)
- Calibration progress update (5-10 min)
- Tutorial update (5 min)
Wins and Celebrations :tada:
Machine status and notices :desktop_computer:
Upcoming meetings and events :spiral_calendar:
- NGEE-Arctic report back (Charlie)
- ESM report back
- ATPESC report back (Greg)
- Exascale climate emulator paper from Keyes lab at KAUST: https://arxiv.org/abs/2408.04440
- AI accelartor (Cerebras) paper: https://dl.acm.org/doi/10.1145/3581784.3627042
- FATES modeling meeting: August 15, Summer Office Hours
- Anne-marie and AGU talk in november
PR board status update
- Dynamic patch arrays: Greg to investiage PVT failure further
- Grass allometry update: crown damage question
- Turnover rate: splitting turnover between canopy and understory question
- Continuum versus discrete canopy/understory: The intention was to allow understory to have better carbon budgets
- Possible literature: https://link.springer.com/chapter/10.1007/978-3-319-27422-5_17
- per_age class discussion:
- We should state fairly clearly what the intention
- Leaf longevity modifications discussion branch diff
- Leaf Layer Photosynthesis Preview
issues and discussions of note
Latest- 2-stream restart error
- Could the maximum number of patches be different between the spin-up and restart?
- Number of canopy layers is set to three for both
- Charlie tested both hybrid mode and finidat-only with no difference
- Could the maximum number of patches be different between the spin-up and restart?
- PVT test not passing
- allvars invalid mem reference
Calibration progress update
- Rosie: objective SP mode calibration work shifting from derecho to betsy (NorESM machine)
- Adrianna: workflow using PFT grids for SP mode targeted to be finished soon.
Tutorial update
- Beta-testing scheduled to start this week. Content continuing to be expanded on.
Walk-on topics
- Ryan: Working on refactoring photosythesis module and adding unit tests
August 5, 2024
- Wins (1 min)
- Machine status and notices (1 min)
- Upcoming meetings (5 min)
- PR board status (5-10 min)
- Latest issues (5 min)
- Calibration progress update (5-10 min)
- Tutorial update (5 min)
Wins and Celebrations :tada:
Machine status and notices :desktop_computer:
Upcoming meetings and events :spiral_calendar:
- FATES modeling meeting: August 15, Summer Office Hours
PR board status update
issues and discussions of note
LatestCalibration progress update
Tutorial update
August 5, 2024
- Wins (1 min)
- Machine status and notices (1 min)
- Lawrencium support (5 min)
- Upcoming meetings (5 min)
- PR board status (5-10 min)
- Use of fates synchronized branches with CTSM (not master)
- Latest issues (5 min)
- Calibration progress update (5-10 min)
- Tutorial update (5 min)
Wins and Celebrations :tada:
Machine status and notices :desktop_computer:
July 29, 2024
- Wins (1 min)
- Machine status and notices (1 min)
- Lawrencium support (5 min)
- Upcoming meetings (5 min)
- PR board status (5-10 min)
- Use of fates synchronized branches with CTSM (not master)
- Latest issues (5 min)
- Calibration progress update (5-10 min)
- Tutorial update (5 min)
Wins and Celebrations :tada:
- Land use v2 E3SM API integrated
Machine status and notices :desktop_computer:
Lawrencium support
Lawrencium moved from CentOS to Rocky8 and the cime configuration to support this machine is broken
- Do we want to prioritize an effort to fix this in the long term?
Upcoming meetings and events :spiral_calendar:
- FATES modeling meeting: August 15, Summer Office Hours
PR board status update
Use of fates synchronized branches with CTSM (not master)
- Motivation: FATES development team can merge in PRs that are synchronized with CTSM on our own schedule, allows users to have a CTSM branch that is always compatible with FATES master
- Questions:
- Should we have a single branch (fates-latest)? Or a branch for each expected tag?
- What frequency should we merge ctsm updates into this (these) branches?
- Every ctsm tag/PR?
- Only when the new branch is merged into ctsm master?
- Should PRs be issued to fates-latest? Or ctsm's master?
issues and discussions of note
LatestCalibration progress update
Tutorial update
July 22, 2024
- Wins (1 min)
- Upcoming meetings (5 min)
- PR board status (5-10 min)
- Photosynthesis Unit Tests (10 min)
- Latest issues (5 min)
- Arctic Shrubs Update (5 min)
- Leaf Temperature Discussion (5 min)
- Calibration progress update (5-10 min)
- Tutorial update (5 min)
Wins and Celebrations :tada:
- NGEE-T proposal review complete
- Land use v2 integrated (E3SM API to be integrated soon)
Upcoming meetings and events :spiral_calendar:
- FATES modeling meeting: August 15, Summer Office Hours
- Pick up regular cadence starting after the tutorial in September?
Machine outages :desktop_computer:
PR board status update
New
Updates
- Dynamic patch arrays: investigating two test run fails
issues and discussions of note
LatestCalibration progress update
Tutorial update
- Beta-testing scheduled for two-weeks starting Aug 12
July 15, 2024
Cancelled due to NGEE-Tropics proposal review
July 8th, 2024
- Wins (1 min)
- Upcoming meetings (5 min)
- PR board status (5-10 min)
- Latest issues (5 min)
- Calibration progress update (5-10 min)
- Tutorial update (5 min)
Wins and Celebrations :tada:
Upcoming meetings and events :spiral_calendar:
- FATES modeling meeting: July 18, Summer Office Hours
Machine outages :desktop_computer:
PR board status update
- No PRs integrated this last week
New
- Time integrated flux diag: needs review
Updates
- Land use v2: integration slated for today post discussion about fb+nocomp mode bareground calculation change
- E3SM land use v2 API: Cleaning up and testing.
- CTSM land use v2 API: Erik to review
- Grazing: reviewed by Sam with some comments/questions. Still needs build/run testing.
issues and discussions of note
Latest- Incorrect per-age-class outputs
- Consensus is that the
_AP
should be weighted by the patch age class area, although we are not consistent currently - Sam also noting that #1170 looks like it may not have fully addressed the issue for
FATES_LAI_AP
.
- Consensus is that the
Calibration progress update
- nothing of note, focus is on preparing for NGEE-T proposal
Tutorial update
- Greg shifting work post landuse v2 to tutorial infrastructure starting this week
July 1st, 2024
- Wins (1 min)
- Upcoming meetings (5 min)
- PR board status (5-10 min)
- Latest issues (5 min)
- Calibration progress update (5-10 min)
- Tutorial update (5 min)
Wins and Celebrations :tada:
- LU V2 seems to be passing tests, ready for integration soon?
Upcoming meetings and events :spiral_calendar:
- FATES modeling meeting: July 18, Summer Office Hours
Machine outages :desktop_computer:
PR board status update
- No PRs integrated this last week
New
- Integrated flux diagnostics - create an issue on the nutrient related bias
- prescribed uptake in the FATES PRT2 test
Reviewers
Updates
- Land use v2: final testing with ctsm and e3sm
- Understory turnover rate: Approved with one minor comment. Needs testing. Has new parameters.
- Plan is to have this be its own API update post land use API update.
- Rosie would like to get this in soon for calibration efforts. As such, we will bring in #1206 and TBD nutrients switch default PR afterwards.
- Dynamic patch arrays: Approved with one minor question. Needs testing.
- Grass allometric equations: Check in with Marcos
- Growth Resp to daily timestep: Jessie provided review
issues and discussions of note
LatestNo new issues or discussions
Calibration progress update
- Rosie: Planning to check in with Charlie about handing off efforts to him wrt land use spin up.
Tutorial update
- FATES tutorial will use FATES API 35
June 24, 2024
- Wins (1 min)
- Upcoming meetings (5 min)
- PR board status (5-10 min)
- Latest issues (5 min)
- Calibration progress update (5-10 min)
- Tutorial update (5 min)
Wins and Celebrations :tada:
- Carbon budget closing thanks to Ryan and Jessie's efforts. PR forthcoming.
Upcoming meetings and events :spiral_calendar:
- FATES modeling meeting: July 18, Summer Office Hours
Machine outages :desktop_computer:
PR board status update
Integrated
- FATES API 35 compat: Merge to next. Waiting integration to master.
Updates
- Land use v2: final testing with ctsm and e3sm
- Understory turnover rate: Approved with one minor comment. Needs testing. Has new parameters.
- Plan is to have this be its own API update post land use API update.
- Rosie would like to get this in soon for calibration efforts. As such, we will bring in #1206 and TBD nutrients switch default PR afterwards.
- Dynamic patch arrays: Approved with one minor question. Needs testing.
- Grass allometric equations: Check in with Xiulin. Needs testing. Updates param values.
- Growth Resp to daily timestep: Ask for another reviewer. Needs build and run testing.
- Jessie will review as well.
issues and discussions of note
LatestNo new issues or discussions
Calibration progress update
- Rosie: Planning to check in with Charlie about handing off efforts to him wrt land use spin up.
Tutorial update
- FATES tutorial will use FATES API 35
June 17, 2024
- Wins (1 min)
- Upcoming meetings (5 min)
- PR board status (5-10 min)
- Latest issues (5 min)
- Carbon balance closure issue
- Calibration progress update (5-10 min)
- Tutorial update (5 min)
Wins and Celebrations :tada:
Upcoming meetings and events :spiral_calendar:
- FATES modeling meeting: June 20, Summer Office Hours
- Graduate student intern visiting NCAR this summer will present briefly on pyrogenic carbon
- Related: Rosie noted that there is a NorESM PR to bring couple trace gasses back to HLM
- Adrianna to send out email
- 2024 CESM workshop debrief
- FATES presentations from Adrianna and Rosie
Machine outages :desktop_computer:
PR board status update
Integrated: History flush and zero fixes
Updates
- Land use v2: correcting precision error in patch splitting
- Understory turnover rate: Tagged Jenny.
- Grass allometric equations: Marcos reviewing. Needs followup?
- FATES API 35 compat: Merge to next. Waiting integration to master.
- Dynamic patch arrays: Greg to review.
- Grazing: Jennifer signed up to review this as well. Timeframe is out beyond land use v2 and may still needs build testing.
issues and discussions of note
Latest- seed production output bug: Jessie has a fix; she will point to her branch to be merged in at next earliest convience.
Calibration progress update
- Discussion of determine the uncertainty of the parameters
Tutorial update
Walk on topics
- Discussion of adapting damage model to fire processes
- Damage model would likely slow down a global model quite a bit
June 10, 2024
- Wins (1 min)
- Upcoming meetings (5 min)
- PR board status (5-10 min)
- Latest issues (5 min)
- Carbon balance closure issue
- Calibration progress update (5-10 min)
- Tutorial update (5 min)
Wins and Celebrations :tada:
Upcoming meetings and events :spiral_calendar:
- FATES modeling meeting: June 20, Summer Office Hours
- Marcos has a possible person to present
- 2024 CESM Workshop: Jun 10 - 13
Machine outages :desktop_computer:
PR board status update
Updates
- History flush and zero fixes: in testing
- Land use v2: Refactoring split_patch to address precision issue in buffer reallocation
- Understory turnover rate: Ask Jenny to review
- Grass allometric equations: Needs review
- FATES API 35 compat: Update to include #1208 and retest
- CTSM Landuse v2 API: Waiting on review. Needs minor deconflict.
- Dynamic patch arrays: initial speed tests look fine, no difference
- Tested with both old and new version to have cap of 3
issues and discussions of note
Latest- default fates_cnp_prescribed_* parameters: to solve, this needs to have prescribed p_uptake and n_uptake turned on for CESM CNP regression test
Carbon balance closure issue
- Marcos: attempt turning off reproduction to determine if that is the issue.
- Ryan/Charlie: NPP minus seed decay should equal allocation
- Ryan: we could also add another data structure to add this up over time
Calibration progress update
no updates
Tutorial update
- Google group for attendees setup
- Greg to send out email with "hello world" docker setup request
June 3, 2024
- Wins (1 min)
- Upcoming meetings (5 min)
- PR board status (5-10 min)
- Latest issues (5 min)
- Calibration progress update (5-10 min)
- Tutorial update (5 min)
Wins and Celebrations :tada:
Upcoming meetings and events :spiral_calendar:
- FATES modeling meeting: June 20, Summer Office Hours
- 2024 CESM Workshop: Jun 10 - 13
Machine outages :desktop_computer:
PR board status update
Updates
- Land use v2: addressing
rsnbl_math_prec
issue during reallocation- Derecho is currently down, which is holding this up.
- History flush and zero fixes: ready for testing
issues and discussions of note
LatestIssues
- migrate fates "classic" projects: github sunsetting "classic" project boards
- PR board migrated. We should triage the others.
- default fates_cnp_prescribed_* parameters: integrate this with #1136.
Discussions
- FATES invetory init: soil initialization followup question
Calibration progress update
Adrianna: updated emulator based on Jessie's suggestions which showed improvement in estimating average GPP.
Tutorial update
May 27, 2024
Cancelled due to Federal Holiday
May 20, 2024
- Wins (1 min)
- Upcoming meetings (5 min)
- PR board status (5-10 min)
- Latest issues (5 min)
- Calibration progress update (5-10 min)
- Tutorial update (5 min)
Wins and Celebrations :tada:
- Greg accepted to ATPESC 2024
Upcoming meetings and events :spiral_calendar:
- FATES modeling meeting: May 23, Shaoqing Liu
- Greg to reach out to Marcos about sending emails
- 2024 CESM Workshop: Jun 10 - 13
Machine outages :desktop_computer:
PR board status update
Integrated: photosytem 2 named constant
Updates
- Land use v2: addressing some run failures
- Ryan: B4B expected
- Growth Resp to daily timestep: review underway
- Understood turnover rate: prioritization?
- Rosie would like to see this in for calibration. Jessie has this on her calirbation work currently.
- Dynamic patch arrays
- Looking for clarification on the consensus from last week
- Agreement that ncl of two is too small.
- Next steps: test performance of ncl of three against dynamic allocation PR
issues and discussions of note
LatestIssues
- arctic shrub pfts
- from Yanlan's up coming paper
- need to think about how to update the hlm-fates pft mapping
- how many do we add for global simulations? And how do we split the fractions?
- duplicate 11, split the fraction, don't use 9
- Long term development: handling divergence in HLM pft data sets
- NaNs in climate driver warning: Erik responded
- carbon budget not closing: discussion deferred
- zero canopy layers no cohorts: discussion deferred
Discussions
- FATES invetory init: Marcos responded
Calibration progress update
- Rosie: need maintresp, allom_d2ca_coeff, allom_d2bl1 to be low to get LAI to observed
- It's reasonable for this to be difficult for the model based on literature research
- Ryan: is radiation model hampering things here? I.e. do we need some sort of explicit gap?
- Adrianna: shared latest updates from variation on results based on number of gridcells included
Tutorial update
May 13, 2024
- Wins (1 min)
- Upcoming meetings (5 min)
- PR board status (5-10 min)
- Latest issues (5 min)
- Calibration progress update (5-10 min)
- Tutorial update (5 min)
Wins and Celebrations :tada:
Upcoming meetings and events :spiral_calendar:
- FATES modeling meeting: May 23, Shaoqing Liu
- No presenters scheduled beyond May 23
- 2024 CESM Workshop: Jun 10 - 13
Machine outages :desktop_computer:
PR board status update
Integrated: Allometry unit test, Inventory input updates, e3sm unstructured grid tool
Updates
- Land use v2
- fraction energy named constant: to be tested ASAP
New
- Dynamic patch arrays
- team formulated to discuss and assess impact of setting
nclmax
- termination mortality proposal for cohorts beyond
nclmax
sounds good
- team formulated to discuss and assess impact of setting
- Growth Resp to daily timestep: needs build testing
- TL;DR: If plants aren't growing, they shouldn't be growth respiring. FATES only grows on daily timestep => only growth respiring on daily timestep
- How does this compare to HLMs?
- Don't use NEP for carbon budgeting; introduce a NBP flux for carbon budgeting.
issues and discussions of note
Latest- no new issues or discussions of note
Calibration progress update
- Adrianna: TRENDY plans discussion
- TRENDY need: External pft-specific seed rain for extinct pfts (for spin up compensation)
- Updated SP mode configurations forthcoming
Tutorial update
- No new updates
May 6, 2024
- Wins (1 min)
- Upcoming meetings (5 min)
- PR board status (5-10 min)
- Latest issues (5 min)
- Calibration progress update (5-10 min)
- Tutorial update (5 min)
Wins and Celebrations :tada:
New group using FATES: Tau Liu at Michigan Tech. Teaming up with Chonggang. Interested in calibration and regional runs.
Upcoming meetings and events :spiral_calendar:
- FATES modeling meeting: 09 May, Rachel Ward
- 2024 E3SM Tutorial: May 7 - 10
- 2024 CESM Workshop: Jun 10 - 13
Machine outages :desktop_computer:
PR board status update
Integrated: CCH water xfer functions, LAI-by-patchage
Updates
- Small fire refactor: ready to test
- Allometry unit testing: testing discussion held on Friday. Minor org refactor to be done.
- Land use v2: Walkthrough held. Break out land use alignment from disturbance TBD.
- CTSM API 36: drafted. run mode combo checks discussed and updated
- How do we test spin up for land use?
- Similar to the fact that we don't have AD testing
- Sam Rabin knows how to add a new test in run_sys_test. Greg will tag up with him later.
- Decision: make issue for testing spin up, but don't hold this up. Add standard regression tests for other landuse v2 modes.
- Deadline: integrated Friday May 24
New
- Land Energy Balance Solve
- what is a good absolute case? Look at distribution of conductance based on different values and come up with reasonable delta.
- Marcos: Stomata have minimum when close so what about using residual as the absolute tolerance?
- Ryan: check in with Gautam for E3SM-side reviewer and feedback
issues and discussions of note
Latest- no new issues or discussions of note
Calibration progress update
- Rosie: One at-a-time ensemble run last week
fates_phen_cold_size_threshold
? See PR #554, grass phenology
Tutorial update
- No new updates
April 29, 2024
- Wins (1 min)
- Upcoming meetings (5 min)
- PR board status (5-10 min)
- Latest issues (5 min)
- Calibration progress update (5-10 min)
- Tutorial update (5 min)
Wins and Celebrations :tada:
- NGEET Phase 3 proposal submitted
- First FATES newsletter sent out
- ELM User's guide PR with connecting FATES documentation
Upcoming meetings and events :spiral_calendar:
- FATES modeling meeting: 09 May, Rachel Ward
- 2024 CESM Workshop
Machine outages :desktop_computer:
PR board status update
- Integrated: E3SM API 33, fire edge case bug fix, day length factor, alternative vert scaling, CTSM API 35
- Updates
- LAI-by-patch-age fix: ready to integrate
- CCH water transfer functions
- Understory turnover rate
issues and discussions of note
Latest- spin-up speed-up
- Dan Ricutto might now
- E3SM specific. Loads stuff into memory.
- growth respiration calc
- Initial solution consideration (Marcos): calculate on a daily basis and apply with each timestep
- Need to have code to account for edge cases given if function based on previous day's value
- We want to get this done prior to the nocomp calibration release
- How does this apply with HLMs (big leaf)?
- Initial solution consideration (Marcos): calculate on a daily basis and apply with each timestep
Calibration progress update
- Adrianna: Daniel and Lydia discussion about setting up emulator. How much do we need for a good set for each pft?
- Rosie: Rerunning one at a time nocomp ensemble with latest main branch (daylength + alt vert scaling).
- Charlie: High seed rain clarification: Rosie wants to dig into this a bit more, ideally with fully spun up case.
- Is there a danger in including seed rain in an ensemble for global calibration?
- Rosie will make global map of seed rain impact
Tutorial update
- Logistics: Organizing travel
Walk-on topics
- Should we re-review this meeting time?
- Goverance meeting?
April 22, 2024
- Wins (1 min)
- Upcoming meetings (5 min)
- Newsletter discussion
- Metrics questions
- PR board status (5-10 min)
- Latest issues (5 min)
- Calibration progress update (5-10 min)
- Tutorial update (5 min)
Wins and Celebrations :tada:
- Greg presented at SEA ISS for US-RSE working group
Upcoming meetings and events :spiral_calendar:
- FATES modeling meeting: April 25, Huanyuan Zhang
- Jessie got confirmation for this week
- Marcos will need someone to host May 23
Newsletter discussion
- What sort of topics would people like to see?
- Add request for fates users
- How to generate content?
- Asynchoronous update + in meeting review
- Dump "wins" from these meetings into the document live
- Ryan to setup calendar reminder for sending it out every quarter
- Rosie: Post-doc position - Rosie
Metrics questions
- How many countries and institutions use FATES?
- Rosie has a pretty good handle on this. She will send out slide with this information.
- US universities is an area of uncertainty.
- Tutorial application could be used for metrics in this regard.
- Rosie has a pretty good handle on this. She will send out slide with this information.
- How many members do we have?
Machine outages :desktop_computer:
PR board status update
- API 35 integration and PR1128 is imminant
- API 33 integration needs confirmation of developer tests passing
- Landuse v2 walkthrough scheduling
- Setup walkthrough meeting next Thursday in the timeslot for fates modeling
- Ryan to reach out to hydro folks for final review
- Adrianna looking for reviews for unit testing and fire refactor
- Unit testing needs cime and shr
- Greg and Charlie to review
issues and discussions of note
Latest- Minor cleanup issue submitted by Sam Rabin
Calibration progress update
- Rosie: shared parameter sensitivity results
Tutorial update
- No new update
Walk-on topics
- Carbon budget issue - DBEN - Jessie
- Post-doc position - Rosie
April 15, 2024
Cancelled
April 8, 2024
- Wins (1 min)
- Upcoming meetings (5 min)
- PR board status (5-10 min)
- Latest issues (5 min)
- Hydro / SP mode compatibility discussion (10 min)
- Calibration progress update (5-10 min)
- Tutorial update (5 min)
Wins and Celebrations :tada:
- Eclipse today! :sun_with_face: :new_moon:
- Perlmutter speed up trick: DVS in read-only mode (see Jessie's email)
- Jennifer's birthday
Upcoming meetings and events :spiral_calendar:
- FATES modeling meeting: 11 Apr 2024, Claire Zarakas
- ESS-Cyberinfrastructure meeting: 15 April 2024
- ESS-PI Meeting: 16-17 April 2024
Machine outages :desktop_computer:
PR board status update
- New allometric modes: Greg testing to check b4b issues Ryan seeing. Test reverting code in EDCanopyStructureMod
- CCH water xfer functions: CCH hydro improvements coming along. Ryan now tracking down issue with non-debug hydro resulting in carbon balance errors.
issues and discussions of note
Latest- duplicate respiration vars: need to remove FATES_AR and similar history outputs
- CSS/PSS file read issue: need a check for missing data and add graceful failure
Calibration progress update
- Rosie: Adrianna and Rosie discussing approach for next steps using Adrianna's pft files (one-at-a-time no comp simulations)
- Rosie to integrate changes from Jessie's vert scaling PR as well
- Workflow discussion
- Jupyter notebooks are not great from a version control perspective, but nicely steps through ensembles updates
Tutorial update
- No new update
Hydro / SP mode compatibility (Ryan)
- Ryan noting that we need to add new features to make hydro and sp mode work together.
- As such, we shouldn't consider failure in the mode to be a bug, but a feature not yet supported.
- Greg to take the action to add graceful failure with both modes in API35 and API34 ctsm and e3sm PRs respectively.
Walk-on topics
- Rosie: Fire BVOC emissions
- Getting MEGAN to work with FATES: https://github.com/NorESMhub/CTSM/issues/36
- Keeping fire emission factors in fates parameter file and push calculations in FATES to HLM
- Rosie currently has a new parameter to map from FATES to MEGAN. Rosie doesn't like that this makes the param file not HLM agnostic.
- Getting MEGAN to work with FATES: https://github.com/NorESMhub/CTSM/issues/36
- Small conference rooms available in B74 (314 and 202)
April 1st, 2024
- Wins (1 min)
- Upcoming meetings (5 min)
- PR board status (5-10 min)
- Latest issues (5 min)
- Recap integration workflow (1 min)
- Calibration progress update (5-10 min)
- Tutorial update (5 min)
Wins and Celebrations :tada:
- Fire trunk fuel bug fix
- Allometry parameters are in!
- Hydro restart bug fix
- FATES tutorial application notices sent out. We've gotten confirmation back from most of them.
Upcoming meetings and events :spiral_calendar:
- FATES modeling meeting: 11 Apr 2024, Claire Zarakas
- Eastern Region Dynamic Global Vegetation Conference: 5-7 April 2024, Woods Hole
Machine outages :desktop_computer:
PR board status update
- New allometric modes: Ryan testing, but running into unexpected non-b4b issues
- Fire refactor: Adrianna to add patch-level weather to this.
issues and discussions of note
Latest- fire trunk issue: fixed per #1180
- Discussion of impacts to current work.
- Team is following up with those that were impacted.
- Adrianna to run coarse global 100 year run to get some assessment of impact. She will update the issue with results and close it out.
- Future refactor is to not include trunk loading in fuel class
- Discussion of impacts to current work.
- SP restart crash: since there is no mort in SP mode, we will avoid these in fix
Calibration progress update
- Jessie: high biomass issue may be alleviated by SP parameters from Adrianna's work. Also recent fire fix may help as well.
- Adrianna: Planning on rerunning with new pft composition to address sparse grid limitations
Tutorial update
see wins
Walk-on topics
- Adrianna: Testing stand structure faster (testing initialization)
- Another option: use the non comp mode with large trees. See https://github.com/NGEET/fates/pull/995 for reference
- SSH security heads up: backdoor found in
xz-utils
version 5.6.0- Fedora Linux 41 users should check security alert notice
March 25, 2024
- Wins (1 min)
- Upcoming meetings (5 min)
- PR board status (5-10 min)
- Latest issues (5 min)
- Improving PR integration workflow discussion
- Calibration progress update (5-10 min)
- Tutorial update (5 min)
Wins and Celebrations :tada:
Upcoming meetings and events :spiral_calendar:
- FATES modeling meeting: 28 Mar 2024, Shaoqing Liu
- Reschedule Rachel?
- Eastern Region Dynamic Global Vegetation Conference: 5-7 April 2024, Woods Hole
Machine outages :desktop_computer:
PR board status update
- Integrated: User controlled history density, Carbon flux recruit history output, hydro ERS, default allom params
- New:
- Updates:
- CCH water xfer functions: Ryan implementing function unit tests
- day length factor: Refactored and tested
Fire refactor overview
issues and discussions of note
Latest- MPI 5x5_amazon resolution on derecho: one
fates
regression test needs update to avoid this - normalize crown area weighted hist
Improving PR integration workflow
- Brainstorming ideas for reducing PR backlogs
- Reviews
- Hold more regular code walk-throughs for large PRs
- Break-up and stage large PRs if possible
- Testing
- Empower core fates modelers to regression test PRs they author
- need: maintaining derecho/perlmutter fates list parity
- Leverage tutorial containers to automate subset of regression tests via github actions
- Related: Automated unit testing
- E3SM tests: https://github.com/E3SM-Project/E3SM/actions/workflows/e3sm-gh-ci-cime-tests.yml
- Empower core fates modelers to regression test PRs they author
- Other logistics
- On-the-fly default parameter file build
- parameter patch file auto-build/checking
Calibration progress update
deferred
Tutorial update
deferred
March 18, 2024
Cancelled
March 11, 2024
- Wins (1 min)
- Upcoming FATES meetings (5 min)
- Weekly synthesis of the PR Status Board (5-10 min)
- Latest issues (5-10 min)
- Calibration progress update (5-10 min)
- Hackathon for fire emissions
Wins and Celebrations :tada:
- Land use V1 integrated into E3SM master
- Landuse data tool hosted on anaconda.org: https://anaconda.org/ngeetropics/tools-fates-landusedata
Upcoming meetings and events :spiral_calendar:
- FATES modeling meeting: 14 Mar 2024, Rachel Ward
- Marcos to host and send out the reminder email
- Eastern Region Dynamic Global Vegetation Conference: 5-7 April 2024, Woods Hole
Machine outages :desktop_computer:
PR board status update
- CCH water xfer functions: resolves #1163, but now seeing 1D hydro failure
- Marcos/Ryan: capping
psi_min
andth_min
discussion?- Discussion with Jenny last week (water balance conservation issue).
- Ryan: Use linear model below zero similar to Van Genukten?
- More discussion needed
- Discussion with Jenny last week (water balance conservation issue).
- Greg to track down regression test failure
- Marcos/Ryan: capping
issues and discussions of note
Latest- hydro error reporting: Greg to handle in the CCH PR
Calibration progress update
- Rosie: discussion of disconnect between tuning and diagnostics packages for land
- Spectral elements runs versus clm diagnostic package
- Jessie: E3SM-unified includes E3SM-diags and ILAMB. Also zppy.
- zppy epic issue: https://github.com/E3SM-Project/zppy/issues/385
- Charlie: are there separate land diagnostics packages?
Fire emission coupling hackathon (Rosie)
- Goal: get fire emissions out of fates and into atmosphere model
- NorESM + Oslo atmo chemistry model
- Calculate biomass burned and pass to HLM to calculate emissions or fates calculate emissions directly
- emissions are pft specific suggesting we should have fates calculate directly
- would result in duplication of emissions factors from HLM
- emissions are pft specific suggesting we should have fates calculate directly
- Charlie: is emission a function of fire "type" (i.e. anthro versus wildfire)
- Rosie: yes, but pft is a proxy of this
- Ryan: discussion or issue?
- Rosie: https://github.com/NGEET/fates/issues/1088 also links to NorESM issue
Hydaulic failure discussion (Chonggang)
- Option: when leaf water potential high, zero stomatal conductance and not request water
- May improve numerical stability
- Chonggang hoping to attempt this at the end of the month
- Ryan: preference is to implement linear model between
- Alternate (short term?): run case at much shorter time steps to get through snags
- Charlie: Develop dry condition test failures (e.g. single site at Atacama)
- Ryan: functional unit testing
- satphen + hydro?
- Greg: being able to parameterize functional unit tests would be great
- Ryan: functional unit testing
Tutorial update
- deferred
March 4, 2024
- Wins (1 min)
- Upcoming FATES meetings (5 min)
- Weekly synthesis of the PR Status Board (5-10 min)
- Latest issues (5-10 min)
- Calibration progress update (5-10 min)
- Global stomatal conductances (5-10 min)
- Tutorial update (5 min)
- Walk-on topics
Wins and Celebrations :tada:
- Ryan's paper was published in JAMES: Nutrient Dynamics in a Coupled Terrestrial Biosphere and Land Model
- Reminder: send out LBL blurb:
Upcoming meetings and events :spiral_calendar:
- CESM Software Engineering Working Group Meeting: Today, 4 March
- FATES modeling meeting: 14 Mar 2024, Rachel Ward
- Marcos to send reminder and will cover the announcment email for Charlie
- Eastern Region Dynamic Global Vegetation Conference: 5-7 April 2024, Woods Hole
- Abstract submission deadline is today
Machine outages :desktop_computer:
PR board status update
- Stuck:
- New category to visually communicate what PRs are held up by bugs found in testing or waiting on upstream fixes
- fates param file autobuild (CTSM#2336): waiting on potential conda fix in #2385
- Leaf phen. with ST3: Greg still investigating
- Update:
- Default allom. params.: Greg to test and integrate. Will have API update with #1128. Make sure updates don't break tests.
- User controlled history density: Ryan updated with requested review changes. To be tested next with API-side update next.
- V2 Landuse Change: Marcos self-assigned review. Outside reviewers post-LMWG meeting?
- Discussion:
- Vertical scaling: Discuss email thread on clarifying scalar definitions and migration to parameter file
- Action:
- Ryan to create new issue for discussion about sla scaling in the biophyscial rates routine
- Jessie: add parameters Ryan suggested to PR to tee up for future implementation without needing to worry about future API update.
issues and discussions of note
Latest- No new issues or updates
Calibration progress update
- No new issues or updates
Global stomatal conductances (Ryan)
- Making coupling HLM-FATES faster. Honing in on the right stomatal conductance.
- Noticing how wide range of values are globally. Why values in the sahara (full fates)?
- Ryan will run an sp mode and share results
Tutorial update
- Application is closed: 73 applicants
- 80% of applicants prefer R
- Greg attending R intro course over the next two weeks
- 80% of applicants prefer R
- tutorial container image build updated to API 31: https://hub.docker.com/r/ngeetropics/elm-builds/tags
- Greg to run a test case this week to ensure current dependencies work with lastest commits
- Related: E3SM forked into NGEET org
- Tutorial build pipeline necessitates downloading HLM tar files associated with tags.
- E3SM doesn't tag merge commits, so I've forked the repo to create our own tags associated with API updates
Walk-on topics
- Rosie: mini hackathon next Thursday/Friday to tackle the fire emissions coupling.
- Add to discussion agenda for next Monday.
February 26, 2024
- Upcoming FATES meetings (5 min)
- FATES software meeting check-in (5 min)
- Weekly synthesis of the PR Status Board (5-10 min)
- Latest issues (5-10 min)
- Calibration progress update (5-10 min)
- Tutorial update (5 min)
- Walk-on topics
Upcoming meetings and events
- FATES modeling meeting: 29 Feb 2024 cancelled due to LMWG
- CESM Land Model & Biogeochem WG meeting: 27-29 Feb 2024
FATES software meeting check-in
- Is there interest in moving this meeting to an every-other week cadence?
- Consensus: those present like the weekly cadence and the reporting back of PRs and issue status.
- Greg: What about the notes format? Would people prefer to engage with the notes in another format?
- Consensus: those present like the transparency of the wiki format and that the notes are co-located on the repo with everything else (issues, discussion, code, etc.)
Machine outages
PR board status update
Updates:
- phosphate fix (E3SM#6222): approved and tested. In the integration queue.
- Leaf phen. with ST3: Greg spent some time with this. Still investigating.
- fates param file autobuild (CTSM#2336): final testing. Next in ctsm queue.
- V2 Landuse Change: Charlie testing. Ready for review. New:
- Day length switch: Greg and Ryan quickly reviewed. Greg to make some minor refactors.
- CCH water transfer func protections: This might help with issue #1157.
- Fire edge-case bugfix: Needs review.
issues and discussions of note
Latest- #1163 and #1165 addressed by PRs #1164 and #1163, respectively
Calibration progress update
- Jessie: finishing up respiration work; will turn into maniscript in near future.
- Considering adoption of Adrianna's ML method for next iteration (post SP-mode calibration updates)
- Ryan: stem respiration for calibration?
- Jessie: Not currently. Kolby keen to investigate this for next project phase. Discussions about field-to-model integration details on going.
Tutorial update
- Jennifer: what's the accessibility goal? Will we explain how to setup the container, etc?
- Jessie: Ultimate goal is to make this available all online with a very simple BCI method.
- Greg: We also have a tutorial setup guide repo in development to host a readthedocs style walkthrough of setting up on a laptop.
- Greg: OLMT experience? We might use or compare the single site subsetting function for pre-processing.
- Jennifer: Yes, two years ago. Current attempts to use were hampered however.
- Ryan: Running pre-process all at once?
- Jessie: yes, we could use the sparse grid NCL method.
- Jessie: Ultimate goal is to make this available all online with a very simple BCI method.
February 12, 2024
- Upcoming FATES meetings (5 min)
- Weekly synthesis of the PR Status Board (5-10 min)
- Latest issues (5-10 min)
- FATES Speed Audit (10 min)
- Calibration progress update (5-10 min)
- Walk-on topics
Upcoming meetings and events
- FATES modeling meeting: 15 Feb 2024, TBD
- CESM Land Model & Biogeochem WG meeting: 27-29 Feb 2024
Machine outages
PR board status update
- Integrated: FATES API refactor (E3SM#6027)
- Updates
- Landuse V1 API (E3SM#5760): Final testing underway.
- New
- Phosphate fix (E3SM#6222)
- Hydro drought decid. restart fix: Fixes #1151. Relation to #1157 below?
- Greg/Ryan to check in with Jenny.
issues and discussions of note
Latest- Hydro restart fail: seems to occur only in drier regions
- Encountered while fixing: #1151
FATES speed audit update (Ryan)
- Can we avoid leaf layer loop: Trimming routine forces us to iterate over leaf layer depth
- Current plan: investigate Ci iteration cap and threshold. Update as necessary along with remembering the previous value.
- Initial iteration analysis update suggests that we often hit the iteration cap.
- Initial iteration testing with ST3 mode (audit was using full dynamics however).
- ED2 experience: What about trying Newton method using the derivitives from reduced analytical form? Review the ED2 approach and share with team.
- Investigate impact of making photosynthesis loosely coupled?
Calibration progress update
- Adrianna: CLM surface data does not seem to quite match ILAMB observation. This is causing issues with gridcell to gridcell comparison.
- Ask Nate Collier for help.
- Calibration team to review Adrianna's brute-force method and will review selected parameter ranges.
February 5, 2024
- Upcoming FATES meetings (5 min)
- Weekly synthesis of the PR Status Board (5-10 min)
- Latest issues (5-10 min)
- Calibration progress update (5-10 min)
- Walk-on topics
Upcoming meetings and events
- FATES performance meeting: Thursday 8 Feb 2024, 1100 PST
- FATES modeling meeting: 15 Feb 2024, TBD
- CESM Land Model & Biogeochem WG meeting: 27-29 Feb 2024
- 2024 FATES tutorial: 9-13 Sept 2024, LBNL
- Informational website up with application form
Machine outages
PR board status update
- Integrated: FATES testmod update (E3SM#6018), Carbon mort reporting, Disable hydro mort., Exp. carbon starv. mort.
- Updates
- New allometric modes: Marcos and Ryan updated. Ready for regression testing.
- User controlled history density: Functionality testing looks good. Ready for regression testing.
- Question: should we split 2d and multiplexed into seperate subroutines? Should we put certain multiplexed into first level or have a combo level?
- Consensus: no strong feelings. This is a good improvement, keep as is. Look at further reorganization as we use it.
- Question: should we split 2d and multiplexed into seperate subroutines? Should we put certain multiplexed into first level or have a combo level?
- New
issues and discussions of note
LatestCalibration progress update
January 29, 2024
- Upcoming FATES meetings (5 min)
- Weekly synthesis of the PR Status Board (5-10 min)
- Latest issues (5-10 min)
- Calibration progress update (5-10 min)
- Walk-on topics
Upcoming meetings and events
- FATES modeling meeting: 01 Feb 2024, TBD
- In lieu of agenda or scheduled speaker, cancel
- CESM Land Model & Biogeochem WG meeting: 27-29 Feb 2024
- Adrianna, Charlie, Greg submitted abstracts. Ryan to submit 2-stream abstract.
Machine outages
PR board status update
- New
- fates param file autobuild: Erik wants to see this come in as initial step.
- Updates
- FATES testmod update (6018): Should be merged to
next
. Waiting to be integrated tomaster
. - Carbon mort reporting: Greg to start testing today.
- New allometric modes: Jessie has some quesitions. Ryan to test and coordinate paramfile updates with #1132.
- Diable hydro mort.: Deconflict discussion against previous #1115
- Leaf phen. with ST3: Not passing exact restart. Investigating.
- User controlled history density: Greg to review today.
- FATES testmod update (6018): Should be merged to
issues and discussions of note
Latest- LU data tool fails on derecho: probably taking too much memory for login node. This will need to be addressed eventually for CLM LU build on the fly.
- This came up in the context of checking ctsm python environment on derecho, due to a dependency issue for xesmf.
- Erik/Charlie: test this on the
develop
queue
Calibration progress update
- Adrianna: Ran Ball-Berry without temp acclimation. Observational mean is more easily achieved, but emulator does a worse job.
- Temperture acclimation without very low vcmax
- Charlie: Claire Zarackas is finding something similar in the absence of hydraulics. So there is a scientific rationale for excluding temp acclimation.
- Temperture acclimation without very low vcmax
- Jessie: met with Ryan to talk about carbon budget issue; add recruitment carbon flux. Carbon budget closes now.
- Not sure if recruitments that die in the same timestep are showing up in the diagnostics
Walk-on topics
- Rosie: Question: Matvey fates model focus? Take over Marius work? Where does it fall in the priority?
- Jennifer: arctic-fire stuff coming up prior to arctic-hydro. Maybe hardening?
- Greg: This is back on API 17, so bringing it up before it gets too stale would be good
- Charlie/Ryan/Rosie: recent hydro issues do suggest that addressing hardening is a good idea.
- Matvey: only calibrated to 2 local sites; will it work globally?
January 22, 2024
- Upcoming FATES meetings (5 min)
- Weekly synthesis of the PR Status Board (5-10 min)
- Building parameter files on the fly (5 min)
- Latest issues (5-10 min)
- Calibration progress update (5-10 min)
- Walk-on topics
Upcoming meetings and events
- FATES modeling meeting: 01 Feb 2024, TBD
Machine outages
PR board status update
- Integrated: Two-stream, stress/drought desc, crown area output, quad solver
- Updates
- FATES testmod update (6018): Threading issue on chrysalis. Greg to update to single threading and restest for the short term.
- Leaf phen. with ST3: Testing today
- Landuse data tool: Refactored tool to merge V1 timeseries tool into V2 LU x pft map tool to share functions
- Jennifer will cherry-pick her daylight factor switch to a new PR branch to be brought main
- Check-in
- Default allom. params.: This is good to go at this point. There is still uncertainty in biomass, but this is a good next step.
- hydro failure disable: Marcos to review Rosie's comments.
- Aside: Rosie noted that Marius' frost work addressed some of this. She and others are writing a proposal to follow up on this.
Building parameter files on the fly
- Ryan to look at moving this forward this week
issues and discussions of note
Latest- hydro dry region failure: Marcos has been working with Jenny on this. Recommendation is that she not trace the issue since its out of scope for her work.
- Ryan proposing to add a fail safe for extreme outlier cases to allow fates run to complete. Still need to make sure this is actually an outlier.
Calibration progress update
- Calibration branch
- Jessie still needs this branch until 2-stream gets to E3SM (probably a few weeks at this point). The calibration branch is pretty close to FATES main for her purposes, so no need to merge main into calibration branch.
Walk-on topics
- Erik asking about status of Marius' frost branch
January 15, 2024
No meeting due to lab holiday
January 8, 2024
- Upcoming FATES meetings (5 min)
- Weekly synthesis of the PR Status Board (5-10 min)
- Latest issues (5-10 min)
- Calibration progress update (5-10 min)
- Walk-on topics
Upcoming meetings and events
- FATES modeling meeting: 18 Jan 2024, TBD
Machine outages
PR board status update
- Integrated: V1 Landuse Change, FatesPFTIndexSwapper update for landuse
- Updates
- Two-stream: Clean up. Erik requesting additional testing. To be discussed on Wed.
- FATES testmod update (6018): Final testing completed. Awaiting review and integration.
- New:
- Carbon Starve Mort: Deconflicted. Needs review.
- Quad eq sovler update: Approved. To be tested.
issues and discussions of note
LatestCalibration progress update
- Jessie presented latest results: GPP and biomass bias improvements
January 1, 2024
No meeting due to lab holiday