Evaluation, Reproducibility, Benchmarks Meeting 38 - Project-MONAI/MONAI GitHub Wiki

Minutes of Meeting 38

Date: 17th September, 2025

Present

  • Rucha
  • Annika
  • Olivier
  • Carole
  • Nick
  • Lena
  • Nicola

Introductions

Welcome, Rucha!

Update from MONAI Advisory Board Meeting

  • There was significant interest in the data licensing project
    • Comment came up that the parent organization is what enforces custom language within existing licenses
  • Any comments about community participation?
    • No, nothing from the advisory board

Update From Licenses Project

  • Is there a tool out there? There are some to explain licenses
  • We could create a tool, but we need to find a legal expert to help, and finding one has been difficult
  • Interesting note -- the parent organization may be playing a role in what license is being used
  • Implications of this work
    • Wake up call for people to be more careful
    • Modeling good practice -- even if we don't have guidelines, we can provide case studies and focus on practical advice
  • Is there a correlation between license use and citations? Industry vs academic use?

Lighthouse Challenge Re-Annotation Project

  • Potential synergy with the MONAI Label team
  • Nick to try to get in touch and have a conversation at MICCAI conference

Potential Collaboration with Other Working Groups

  • Ultrasound group (new) expressed interest, but it is not a top priority on their side just yet. Annika will update us if this changes
  • Potential synergy with federated learning working group -- Annika/Carole to reach out and just start a conversation

Updates from Olivier/Confidence Interval Project

  • Background
    • There is poor reporting of confidence intervals in our community
    • This is at odds with regulatory needs, and also just a bad scientific practice in general
      • Could help to explain limited translation
    • Lack of CIs also leads to unjustified claims of superiority in new methods
    • Goal of the project
      • Point out the problem
      • Present experimental results in medical imaging work specifically
      • Not guidelines yet, but headed in that direction
  • Update
    • Finished additional experiments this summer
    • Working on journal article that expands on workshop paper
    • Each "benchmark instance" is one team's performance on one particular task (in decathlon)
  • Plan
    • Want to send this to Medical Image Analysis

Thoughts for Next Time

  • How do we get more developers contributing?
    • Unclear how much is done by core developers vs community contributions
  • Next meeting October 22