Approving Production Runs - HopkinsIDD/cholera-mapping-pipeline GitHub Wiki

This procedure refers to the process of approving model runs. By this stage, the Stan model should have successfully completed and diagnostic reports generated for the model run.

Diagnostic Reports Required:

• country data report
• data comparison report
• generated quantities report (output summary report)

Individuals reviewing the diagnostic reports should add comments to the country-named issue in the appropriate Kanban board. The comments should include an evaluation of the model fit and convergence and propose potential reasons for unusual things that may be observed in the reports. The comments should also include a "vote" (individual-level determination about whether the run should be approved) and proposed action items if the run is not approved.

Here is an example review that an individual may submit:

Review: Model fits and convergence look pretty good. However, there is notable over-estimation in two censored observations from 2017-2019 in OC 201579. The reason for this over-estimation may be a large discrepancy between OC 201579 and the WHO data, even for the national level observation. BDI reported >1000 cases to WHO in 2019 and the observation in OC 2019 is censored in 2019 (covers Jan to July 2019) and reports close to 0 cases. The trade-off between censoring in the observation with a low case count and a much larger complete observation may have resulted in an inflated case estimate.

Vote: approve conditional on generated quantities report
Proposed follow-up: merge dev_output_stats branch, run generated quantities, and generate output stats report

Model runs may only be approved if three individuals review the diagnostic reports and vote to approve the run.