Evaluation, Reproducibility, Benchmarks Meeting 15 - Project-MONAI/MONAI GitHub Wiki

Minutes of meeting 15

Date: 23rd February 2022

Present: Nicola, Carole, Keyvan, Lena, Annika

TOP 1: MONAI metric implementation

*Delphi recommendation framework: List of metrics including reference implementation (desired: MONAI implementation for all metrics ready by paper submission)

  • Provide list of metrics to MONAI developers
    • We need to identify how the metrics can be implemented and the optimal implementation
    • โ†’ Take this to the expert groups: Carole will formulate the required questions. We should ask them:
      • Ambiguities in the implementation
      • Reference implementation
      • Reference publication
      • If a metric can be implemented in multiple ways: Which ones? Which one is preferable?
      • Metric prioritization
    • Idea: Write follow-up paper with implementation details, lead by Carole

TOP 2: Semantic segmentation metric mapping

  • Question: How should we deal with lots of text?
  • For the toolkit it is preferable to provide all discussion points
  • For paper: Refer to paragraph/additional figure/table
  • Main paper: Only include the meta road
    • One figure with the complete process: Generate FP โ†’ Identify problem category โ†’ Problem category specific meta roads (keep it simple, e.g. โ€œSelect overlap metric (if any) and only keep very straightforward sub-processes (e.g. volume)) โ†’ Branches come together again for aggregation