Investigating model execution failure in Dev and Prod ‐ Feb 2026 - conrad-blucher-institute/semaphore GitHub Wiki
created from this google doc
Semaphore Dev VS Prod
This report analyzes model execution failures in Semaphore Dev and Prod from February 3, 2026 12:00am - February 4, 2026 ~2:00pm. Both environments show similar overall success rates (96.64% Dev, 96.54% Prod), but exhibit distinct failure patterns that require attention.
Key Findings
- Magnolia Transform models still failing
- Stale SEMAPHORE prediction data is the primary failure cause
- Production shows more missing data failures than Development (24 vs 9)
- NDFD weather forecast staleness affects both environments equally
Overall Statistics
| Metric | Development | Production |
|---|---|---|
| Total Executions | 4,559 | 4,623 |
| Successful | 4,406 | 4,463 |
| Failed | 153 | 160 |
| Success Rate | 96.64% | 96.54% |
| Failure Rate | 3.36% | 3.46% |
Both environments maintain high success rates above 96%, with Prod showing a marginally higher failure rate (0.10% difference). This is likely due to the fact that I ran my analysis scripts 20 minutes apart and thus Prod had more runs and more failures overall compared to Dev.
Failure Reasons Breakdown
| Reason | Dev Count | Prod Count | Difference |
|---|---|---|---|
| Stale Data | 144 (94%) | 136 (85%) | +8 Dev |
| Missing Data | 9 (6%) | 24 (15%) | +15 Prod |
Failures by Data Source
| Data Source | Development | Production |
|---|---|---|
| SEMAPHORE | 86 | 94 |
| NDFD_JSON (Weather) | 58 | 58 |
| NOAATANDC (Tides) | 9 | 8 |
Key Observations:
- SEMAPHORE internal predictions are the primary failure source in both environments
- NDFD weather forecast staleness is identical (58 failures)
- NOAA tide data shows issues across both environments
Guys we’re so silly:
The Magnolia Transform models represent the most significant reliability issue, accounting for 56% of Development failures and 59% of Production failures.
| Model | Dev Failures | Prod Failures | Primary Issue |
|---|---|---|---|
| magnolia_transform_24 | 43 | 43 | Stale SEMAPHORE data (12hr) |
| magnolia_transform_48 | 43 | 43 | Stale SEMAPHORE data (12hr) |
| magnolia_transform_12 | 0 | 8 | Missing SEMAPHORE data (PROD ONLY) |
| Total Magnolia Failures | 86 | 94 |
Root Cause Analysis:
Magnolia Transform models depend on upstream SEMAPHORE surge predictions. When these predictions become stale (>12 hours old) the transform models cannot execute.
Currently the transform models use the default staleness value which is 7, meaning that they never have an unstale series to ingest because the first magnolia model takes at least 24 hours to produce enough predictions for the transform models to use. I’m honestly not sure how the magnolia 12 model is running successfully on dev with this in mind.
The reason we didn’t catch this immediately is because the discord notifications don’t separate between missing data and stale data.
I’ve created a ticket here to correct this issue.
ExternalData Sources Issues
-
Address NDFD Weather Forecast Staleness
NDFD_JSON weather forecasts cause 58 failures identically across both environments:
- This is because we go over the staleness threshold by give or take 33 minutes (10:33:32).
- I think we should increase the threshold to 11.
- Monitor NOAA
| 02/04/26 02:22:11: DateRangeValidation: Series \[Series\] \-\> description: \[SeriesDescription\] \-\> source: NOAATANDC, series: dWl, location: VirginiaKey, datum: NAVD, dataIntegrityDescription: None, verificationOverride: None, timeDescription: \[TimeDescription\] \-\> fromDateTime: 2026-02-02 02:00:00+00:00, toDateTime: 2026-02-04 02:00:00+00:00. interval: 1:00:00, stalenessOffset: None is missing 1 values. 02/04/26 02:22:11: Missing time: 2026-02-04 02:00:00+00:00 02/04/26 02:22:11: Warning:: Prediction failed due to lack of data 02/04/26 02:22:11: Exception message: Error Code: \-1 DateRangeValidation Failed in Data Gatherer\! \[Series\] \-\> \[Series\] \-\> description: \[SeriesDescription\] \-\> source: NOAATANDC, series: dWl, location: VirginiaKey, datum: NAVD, dataIntegrityDescription: None, verificationOverride: None, timeDescription: \[TimeDescription\] \-\> fromDateTime: 2026-02-02 02:00:00+00:00, toDateTime: 2026-02-04 02:00:00+00:00. interval: 1:00:00, stalenessOffset: None 02/04/26 02:22:11: Full stack trace: Traceback (most recent call last): File "/app/src/orchestrator.py", line 79, in run\_semaphore data\_repository \= self.dataGatherer.get\_data\_repository(DSPEC, reference\_time) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/src/ModelExecution/dataGatherer.py", line 60, in get\_data\_repository dependent\_data\_repository \= self.\_\_request\_dependent\_data(dependentSeries, referenceTime) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/src/ModelExecution/dataGatherer.py", line 123, in \_\_request\_dependent\_data self.\_\_validate\_series(series, referenceTime) File "/app/src/ModelExecution/dataGatherer.py", line 233, in \_\_validate\_series raise Semaphore\_Data\_Exception(f'DateRangeValidation Failed in Data Gatherer\! \\n\[Series\] \-\> {series}') exceptions.Semaphore\_Data\_Exception: Error Code: \-1 DateRangeValidation Failed in Data Gatherer\! \[Series\] \-\> \[Series\] \-\> description: \[SeriesDescription\] \-\> source: NOAATANDC, series: dWl, location: VirginiaKey, datum: NAVD, dataIntegrityDescription: None, verificationOverride: None, timeDescription: \[TimeDescription\] \-\> fromDateTime: 2026-02-02 02:00:00+00:00, toDateTime: 2026-02-04 02:00:00+00:00. interval: 1:00:00, stalenessOffset: None 02/04/26 02:22:11: Model 48hr\_VirginiaKey\_wl FAILED \- Null result inserted 02/04/26 02:22:11: Failed results: \[Series\] \-\> description: \[SemaphoreSeriesDescription\] \-\> AIName: 48hr\_VirginiaKey\_wl, AIGeneratedVersion: 1.0.0, DataSeries: pWl, Location: VirginiaKey, Datum: NAVD, timeDescription: None 02/04/26 02:22:11: dataValue dataUnit timeGenerated leadTime 0 None meter 2026-02-04 02:00:00+00:00 2 days |
| :---- |
- It seems like all of the failures follow this kind of format where we’re missing one datapoint for Virginia Key models.
- None of the NoData errors since pushing though, which is a plus!
All Models Table
| Model Name | Total Runs | Successes | Failures | Success % | Most Common Failure |
|---|---|---|---|---|---|
| magnolia_transform_24 | 86 | 0 | 86 | 0.0% | stale_data (SEMAPHORE) |
| magnolia_transform_48 | 86 | 0 | 86 | 0.0% | stale_data (SEMAPHORE) |
| magnolia_transform_12 | 86 | 78 | 8 | 90.7% | missing_data (SEMAPHORE) |
| 96hr_VirginiaKey_wl | 86 | 82 | 4 | 95.35% | missing_data (NOAATANDC) |
| Bird-Island_Water-Temperature_30hr | 86 | 82 | 4 | 95.35% | stale_data (NDFD_JSON) |
| Bird-Island_Water-Temperature_42hr | 86 | 82 | 4 | 95.35% | stale_data (NDFD_JSON) |
| Bird-Island_Water-Temperature_36hr | 86 | 82 | 4 | 95.35% | stale_data (NDFD_JSON) |
| Bird-Island_Water-Temperature_3hr | 86 | 82 | 4 | 95.35% | stale_data (NDFD_JSON) |
| Bird-Island_Water-Temperature_54hr | 86 | 82 | 4 | 95.35% | stale_data (NDFD_JSON) |
| Bird-Island_Water-Temperature_6hr | 86 | 82 | 4 | 95.35% | stale_data (NDFD_JSON) |
| Bird-Island_Water-Temperature_48hr | 86 | 82 | 4 | 95.35% | stale_data (NDFD_JSON) |
| Bird-Island_Water-Temperature_12hr | 86 | 82 | 4 | 95.35% | stale_data (NDFD_JSON) |
| Bird-Island_Water-Temperature_18hr | 86 | 82 | 4 | 95.35% | stale_data (NDFD_JSON) |
| Bird-Island_Water-Temperature_24hr | 86 | 82 | 4 | 95.35% | stale_data (NDFD_JSON) |
| Bird-Island_Water-Temperature_60hr | 87 | 83 | 4 | 95.4% | stale_data (NDFD_JSON) |
| Bird-Island_Water-Temperature_108hr | 87 | 83 | 4 | 95.4% | stale_data (NDFD_JSON) |
| Bird-Island_Water-Temperature_90hr | 87 | 83 | 4 | 95.4% | stale_data (NDFD_JSON) |
| Bird-Island_Water-Temperature_96hr | 87 | 83 | 4 | 95.4% | stale_data (NDFD_JSON) |
| Bird-Island_Water-Temperature_78hr | 87 | 83 | 4 | 95.4% | stale_data (NDFD_JSON) |
| Bird-Island_Water-Temperature_84hr | 87 | 83 | 4 | 95.4% | stale_data (NDFD_JSON) |
| Bird-Island_Water-Temperature_66hr | 87 | 83 | 4 | 95.4% | stale_data (NDFD_JSON) |
| Bird-Island_Water-Temperature_72hr | 87 | 83 | 4 | 95.4% | stale_data (NDFD_JSON) |
| rp_S2S_surge_72h | 87 | 83 | 4 | 95.4% | stale_data (NDFD_JSON) |
| rp_S2S_surge_48h | 87 | 83 | 4 | 95.4% | stale_data (NDFD_JSON) |
| rp_S2S_surge_12h | 87 | 83 | 4 | 95.4% | stale_data (NDFD_JSON) |
| ar_mlp_surge_24h | 87 | 83 | 4 | 95.4% | stale_data (NDFD_JSON) |
| ar_mlp_surge_12h | 87 | 83 | 4 | 95.4% | stale_data (NDFD_JSON) |
| ar_mlp_surge_48h | 87 | 83 | 4 | 95.4% | stale_data (NDFD_JSON) |
| ar_mlp_surge_72h | 87 | 83 | 4 | 95.4% | stale_data (NDFD_JSON) |
| Bird-Island_Water-Temperature_102hr | 87 | 83 | 4 | 95.4% | stale_data (NDFD_JSON) |
| Bird-Island_Water-Temperature_114hr | 87 | 83 | 4 | 95.4% | stale_data (NDFD_JSON) |
| Bird-Island_Water-Temperature_120hr | 87 | 83 | 4 | 95.4% | stale_data (NDFD_JSON) |
| rp_S2S_surge_24h | 87 | 83 | 4 | 95.4% | stale_data (NDFD_JSON) |
| nj_mlp_surge_72h | 87 | 84 | 3 | 96.55% | missing_data (NOAATANDC) |
| 48hr_VirginiaKey_wl | 86 | 84 | 2 | 97.67% | missing_data (NOAATANDC) |
| 72hr_VirginiaKey_wl | 86 | 84 | 2 | 97.67% | missing_data (NOAATANDC) |
| ar_inundation_apr_48h | 87 | 85 | 2 | 97.7% | missing_data (NOAATANDC) |
| nj_mlp_surge_48h | 87 | 85 | 2 | 97.7% | missing_data (NOAATANDC) |
| nj_mlp_surge_24h | 87 | 85 | 2 | 97.7% | missing_data (NOAATANDC) |
| MRE_Bird-Island_Water-Temperature_24hr | 87 | 87 | 0 | 100.0% | No failures |
| MRE_Bird-Island_Water-Temperature_18hr | 87 | 87 | 0 | 100.0% | No failures |
| MRE_Bird-Island_Water-Temperature_6hr | 87 | 87 | 0 | 100.0% | No failures |
| MRE_Bird-Island_Water-Temperature_72hr | 87 | 87 | 0 | 100.0% | No failures |
| MRE_Bird-Island_Water-Temperature_78hr | 87 | 87 | 0 | 100.0% | No failures |
| MRE_Bird-Island_Water-Temperature_84hr | 87 | 87 | 0 | 100.0% | No failures |
| MRE_Bird-Island_Water-Temperature_90hr | 87 | 87 | 0 | 100.0% | No failures |
| MRE_Bird-Island_Water-Temperature_96hr | 87 | 87 | 0 | 100.0% | No failures |
| MRE_Bird-Island_Water-Temperature_60hr | 87 | 87 | 0 | 100.0% | No failures |
| ThermalRefuge | 88 | 88 | 0 | 100.0% | No failures |
| ar_inundation_apr_12h | 87 | 87 | 0 | 100.0% | No failures |
| MRE_Bird-Island_Water-Temperature_36hr | 87 | 87 | 0 | 100.0% | No failures |
| MRE_Bird-Island_Water-Temperature_30hr | 87 | 87 | 0 | 100.0% | No failures |
| MRE_Bird-Island_Water-Temperature_42hr | 87 | 87 | 0 | 100.0% | No failures |
| MRE_Bird-Island_Water-Temperature_48hr | 87 | 87 | 0 | 100.0% | No failures |
| MRE_Bird-Island_Water-Temperature_54hr | 87 | 87 | 0 | 100.0% | No failures |
| MRE_Bird-Island_Water-Temperature_3hr | 87 | 87 | 0 | 100.0% | No failures |
| MRE_Bird-Island_Water-Temperature_66hr | 87 | 87 | 0 | 100.0% | No failures |
| 12hr_VirginiaKey_wl | 86 | 86 | 0 | 100.0% | No failures |
| MRE_Bird-Island_Water-Temperature_102hr | 87 | 87 | 0 | 100.0% | No failures |
| 24hr_VirginiaKey_wl | 86 | 86 | 0 | 100.0% | No failures |
| MRE_Bird-Island_Water-Temperature_114hr | 87 | 87 | 0 | 100.0% | No failures |
| MRE_Bird-Island_Water-Temperature_120hr | 87 | 87 | 0 | 100.0% | No failures |
| MRE_Bird-Island_Water-Temperature_12hr | 87 | 87 | 0 | 100.0% | No failures |
| MRE_Bird-Island_Water-Temperature_108hr | 87 | 87 | 0 | 100.0% | No failures |
| ar_inundation_jul_12h | 86 | 86 | 0 | 100.0% | No failures |
| ar_inundation_jan_48h | 87 | 87 | 0 | 100.0% | No failures |
| ar_inundation_jan_24h | 87 | 87 | 0 | 100.0% | No failures |
| ar_inundation_jan_12h | 87 | 87 | 0 | 100.0% | No failures |
| ar_inundation_feb_48h | 87 | 87 | 0 | 100.0% | No failures |
| ar_inundation_feb_24h | 87 | 87 | 0 | 100.0% | No failures |
| ar_inundation_feb_12h | 87 | 87 | 0 | 100.0% | No failures |
| ar_inundation_dec_48h | 87 | 87 | 0 | 100.0% | No failures |
| ar_inundation_dec_24h | 87 | 87 | 0 | 100.0% | No failures |
| ar_inundation_dec_12h | 87 | 87 | 0 | 100.0% | No failures |
| ar_inundation_aug_48h | 87 | 87 | 0 | 100.0% | No failures |
| ar_inundation_aug_24h | 87 | 87 | 0 | 100.0% | No failures |
| ar_inundation_aug_12h | 87 | 87 | 0 | 100.0% | No failures |
| ar_inundation_apr_24h | 87 | 87 | 0 | 100.0% | No failures |
| ar_inundation_jul_24h | 86 | 86 | 0 | 100.0% | No failures |
| ar_inundation_jul_48h | 86 | 86 | 0 | 100.0% | No failures |
| ar_inundation_oct_24h | 86 | 86 | 0 | 100.0% | No failures |
| ar_inundation_oct_12h | 86 | 86 | 0 | 100.0% | No failures |
| ar_inundation_nov_48h | 86 | 86 | 0 | 100.0% | No failures |
| ar_inundation_nov_24h | 86 | 86 | 0 | 100.0% | No failures |
| ar_inundation_nov_12h | 86 | 86 | 0 | 100.0% | No failures |
| ar_inundation_may_48h | 86 | 86 | 0 | 100.0% | No failures |
| ar_inundation_may_24h | 86 | 86 | 0 | 100.0% | No failures |
| ar_inundation_may_12h | 86 | 86 | 0 | 100.0% | No failures |
| ar_inundation_oct_48h | 86 | 86 | 0 | 100.0% | No failures |
| ar_inundation_sep_12h | 86 | 86 | 0 | 100.0% | No failures |
| ar_inundation_jun_12h | 86 | 86 | 0 | 100.0% | No failures |
| ar_inundation_jun_24h | 86 | 86 | 0 | 100.0% | No failures |
| ar_inundation_jun_48h | 86 | 86 | 0 | 100.0% | No failures |
| ar_inundation_mar_12h | 86 | 86 | 0 | 100.0% | No failures |
| ar_inundation_mar_24h | 86 | 86 | 0 | 100.0% | No failures |
| ar_inundation_mar_48h | 86 | 86 | 0 | 100.0% | No failures |
| nj_mlp_surge_12h | 87 | 87 | 0 | 100.0% | No failures |
| magnolia_12 | 86 | 86 | 0 | 100.0% | No failures |
| magnolia_24 | 86 | 86 | 0 | 100.0% | No failures |
| magnolia_48 | 87 | 87 | 0 | 100.0% | No failures |
| ar_inundation_sep_48h | 86 | 86 | 0 | 100.0% | No failures |
| ar_inundation_sep_24h | 86 | 86 | 0 | 100.0% | No failures |
| pi_mlp_surge_24h | 87 | 87 | 0 | 100.0% | No failures |
| pi_mlp_surge_12h | 87 | 87 | 0 | 100.0% | No failures |
| pi_mlp_surge_48h | 87 | 87 | 0 | 100.0% | No failures |
| pi_mlp_surge_72h | 87 | 87 | 0 | 100.0% | No failures |