Investigating Model Execution Failures In Dev February 7‐9, 2026 - conrad-blucher-institute/semaphore GitHub Wiki
Semaphore Development Environment Analysis
Created from this google doc: dev_analysis_02-09-2026
February 7-9, 2026
Analysis Period: February 7, 2026 12:00 AM - February 9, 2026 5:00 PM
Executive Summary
The development environment shows a 97.85% success rate across 6,777 model executions, with 146 failures requiring attention. The analysis reveals three critical issues:
- MAGNOLIA TRANSFORM CRISIS: magnolia_transform_48 has a 0% success rate (65/65 failures)
- LIGHTHOUSE DATA SOURCE: New failure source causing 44% of all failures
- SEMAPHORE INTERNAL PREDICTIONS: Missing data affecting transform models
Overall Statistics
| Total Executions | 6,777 |
|---|---|
| Successful | 6,631 |
| Failed | 146 |
| Success Rate | 97.85% |
| Failure Rate | 2.15% |
Failure Analysis
Failure Reasons Breakdown
| missing_data | 98 (67%) |
|---|---|
| stale_data | 48 (33%) |
Failures by Data Source
| LIGHTHOUSE | 64 (44%) |
|---|---|
| SEMAPHORE | 54 (37%) |
| NOAATANDC | 27 (18%) |
| NDFD_JSON | 1 (<1%) |
Models with LIGHTHOUSE or NOAATANDC failures.
| Model Name | Runs | Success % | Fails | Data Source(s) | Failure Reason(s) |
|---|---|---|---|---|---|
| magnolia_transform_48 | 65 | 0.0% | 65 | LIGHTHOUSE (47, 72%), SEMAPHORE (18, 28%) | stale_data (47, 72%), missing_data (18, 28%) |
| magnolia_transform_12 | 65 | 50.77% | 32 | SEMAPHORE (18, 56%), LIGHTHOUSE (14, 44%) | missing_data (32, 100%) |
| 96hr_VirginiaKey_wl | 65 | 93.85% | 4 | NOAATANDC (3, 75%), NDFD_JSON (1, 25%) | missing_data (3, 75%), stale_data (1, 25%) |
| 72hr_VirginiaKey_wl | 65 | 95.38% | 3 | NOAATANDC (3, 100%) | missing_data (3, 100%) |
| ar_inundation_apr_24h | 65 | 96.92% | 2 | NOAATANDC (2, 100%) | missing_data (2, 100%) |
| pi_mlp_surge_48h | 65 | 96.92% | 2 | NOAATANDC (2, 100%) | missing_data (2, 100%) |
| ar_inundation_apr_48h | 65 | 96.92% | 2 | NOAATANDC (2, 100%) | missing_data (2, 100%) |
| ar_inundation_apr_12h | 65 | 96.92% | 2 | NOAATANDC (2, 100%) | missing_data (2, 100%) |
| rp_S2S_surge_72h | 64 | 96.88% | 2 | NOAATANDC (2, 100%) | missing_data (2, 100%) |
| rp_S2S_surge_48h | 65 | 96.92% | 2 | NOAATANDC (2, 100%) | missing_data (2, 100%) |
| pi_mlp_surge_72h | 65 | 96.92% | 2 | NOAATANDC (2, 100%) | missing_data (2, 100%) |
| 48hr_VirginiaKey_wl | 64 | 98.44% | 1 | NOAATANDC (1, 100%) | missing_data (1, 100%) |
| magnolia_12 | 64 | 98.44% | 1 | LIGHTHOUSE (1, 100%) | missing_data (1, 100%) |
| nj_mlp_surge_24h | 44 | 97.73% | 1 | NOAATANDC (1, 100%) | missing_data (1, 100%) |
| nj_mlp_surge_12h | 44 | 97.73% | 1 | NOAATANDC (1, 100%) | missing_data (1, 100%) |
| magnolia_24 | 64 | 98.44% | 1 | LIGHTHOUSE (1, 100%) | missing_data (1, 100%) |
| magnolia_48 | 64 | 98.44% | 1 | LIGHTHOUSE (1, 100%) | missing_data (1, 100%) |
| pi_mlp_surge_24h | 65 | 98.46% | 1 | NOAATANDC (1, 100%) | missing_data (1, 100%) |
| pi_mlp_surge_12h | 65 | 98.46% | 1 | NOAATANDC (1, 100%) | missing_data (1, 100%) |
| nj_mlp_surge_72h | 44 | 97.73% | 1 | NOAATANDC (1, 100%) | missing_data (1, 100%) |
| nj_mlp_surge_48h | 44 | 97.73% | 1 | NOAATANDC (1, 100%) | missing_data (1, 100%) |
Critical Issues & Recommended Tickets
ISSUE #1: Magnolia Transform Models
Impact: 115 failures (78.8% of all failures)
magnolia_transform_48
- Status: 0% success rate (0/65 successful)
- Primary Issue: Stale LIGHTHOUSE data (47) + Missing SEMAPHORE data (18)
magnolia_transform_12
- Status: 50.8% success rate (33/65 successful)
- Primary Issue: Missing SEMAPHORE (18) + LIGHTHOUSE (14)
magnolia_transform_24
- Status: 72.3% success rate (47/65 successful)
- Primary Issue: Missing SEMAPHORE data (18)
Recommended Tickets
TICKET 1: Investigate LIGHTHOUSE Data Source
- Impact: 64 failures (44% of total)
- What is going on with thissssss 🎶
- Three potential fixes
- 1. Rethink Absolute value… nah -> add comment about britalness
- 2. Change lighthouse to set generated time to now time for predicted values
TICKET 2: Fix Magnolia Transform LIGHTHOUSE Staleness
- Impact: magnolia_transform_48 has 0% success rate
- Action: Review and adjust staleness thresholds for LIGHTHOUSE inputs
TICKET 3: Investigate Magnolia Transform Missing SEMAPHORE Data
- Impact: 54 failures across all transform models
- I fear this is due to Lighthouse issues for the base magnolia models, ticket below. Whenever the base models have one failure we have to wait until there's a full uninterrupted 12 hour run before we can run the transforms again.
TICKET 4: Monitor LIGHTHOUSE Integration for Base Magnolia Models
- Impact: 3 failures (1 per base magnolia model)
TICKET 5: Investigate NOAATANDC 01:00 UTC Data Gap
- Impact: 27 failures, primarily at 01:00 UTC
Key Insights
- NDFD issue from previous report is FIXED - only 1 failure (was 58)
- Overall system reliability improved +1.21% compared to previous analysis
- LIGHTHOUSE is a new data source requiring immediate attention
- Failure type shifted from stale_data (94%) to missing_data (67%)
- 84 models maintain perfect 100% success rate
Summary
The development environment shows improved overall performance with excellent resolution of the NDFD staleness issue. However, LIGHTHOUSE integration and magnolia_transform models require immediate attention. The shift from stale_data to missing_data suggests threshold fixes revealed underlying data availability problems.