Investigating Model Execution Failures In Dev 02 18 2026 - conrad-blucher-institute/semaphore GitHub Wiki

Based off of this document: Investigating Model Execution Failures In Dev 02/18/2026

Model Failure Analysis

Feb 8 – Feb 18, 2026 · 11 days · 106 models · sherlock-dev

OVERVIEW

TOTAL RUNS 26,346 106 models × ~249 hrs	FAILURES 1,751 6.6% failure rate	MISSING DATA 1,655 94.5% of all failures	STALE DATA 96 5.5% of all failures

Nearly all failures are missing data failures, not stale data. The failures break down across four sources: LIGHTHOUSE (most impactful), SEMAPHORE (a dependency cascade from LIGHTHOUSE), NOAATANDC, and NDFD_JSON (stale only).

TLDR:

LIGHTHOUSE ISSUES: All Bird Island and All Magnolia.

For Bird Island models are failing because of a missing point in LIGHTHOUSE falling on the edges of the interpolation window. The proposed solution is to request a larger window than needed.

For Magnolia models are failing more consistently due to missing points in LIGHTHOUSE data because we do not interpolate on wind speed or wind direction. The points themselves are missing in LIGHTHOUSE and as far as I know are not recoverable. The proposed solution is to interpolate on these series if possible.

SEMAPHORE DATA ISSUES:

Only a symptom of LIGHTHOUSE issues.

NOAATANDC ISSUES:

Small amount of missing points around the same time frame, maybe an update window, needs further investigation.

NDFD ISSUES:

Likely an uncommon case, they didn’t update for like 13 hours on February 10th, but no further issues afterward.

DAILY FAILURE RATE TREND

Failure rate by day (%)

There is no improving trend — failures remain volatile across the full period, ranging from 1.4% (Feb 8) to 11.6% (Feb 14). The relatively clean days (Feb 8, 11, 15) correspond to days with fewer or shorter LIGHTHOUSE gaps.

FAILURES BY DATA SOURCE

SOURCE	TYPE	MISSING FAILURES	STALE	MODELS AFFECTED	NOTES
LIGHTHOUSE	missing	905	—	47	Intermittent hourly gaps; cascades to all dependents
SEMAPHORE	missing	678	—	3	magnolia_transform cascade from failed magnolia output
NOAATANDC	missing	72	—	16	Scattered gaps; clustering near 00-01 UTC
NDFD_JSON	stale	—	62	29	Single event: Feb 10 05-06 UTC only

LIGHTHOUSE — MISSING DATA ANALYSIS

LIGHTHOUSE is the dominant failure driver. It causes two distinct failure patterns: major outages where a gap timestamp triggers failures across all LIGHTHOUSE-dependent models simultaneously, and minor gaps that only affect the 3 magnolia models (which have a shorter lookback requirement).

	Key finding: LIGHTHOUSE failures are not time-of-day patterned The missing data timestamps are distributed across all hours with no clustering around any particular time window. These appear to be random, non-recurring outages rather than a scheduled maintenance window or predictable refresh lag.

Table of all lighthouse failures: lighthouse_flat.docx

Almost every gap is just 1 missing point — a single hourly LIGHTHOUSE timestamp that never arrived. Because there are no time trends, I don’t think that this is because of any delays in updating points or But because nothing ever goes back and fills it in, every model run for the next 24 hours keeps checking, finding the same missing slot, and failing again. The gap itself might have lasted minutes, but the damage lasts a full day. So rather than a LIGHTHOUSE reliability problem we’re actually dealing with a backfilling problem (how relevant to the water quality team ☠️).

I wanted to look at whether LIGHTHOUSE ever gets these missing points at all. If we query the database for those specific gap timestamps at the locations that the models ask for do those rows exist now? If they do, the problem lies somewhere in semaphore’s ingestion, if they don't, the data was genuinely lost and never recovered, which is a different problem.

These are the timestamps missing on February 10th. The Magnolia models get their data from the Port Lavaca Station and Port O'Connor Station, both Bird Island models get their data from the South Bird Island Station.

Tuesday, February 10
02/10/26 00:22	magnolia_48	2026-02-09 18:00 UTC	1
02/10/26 00:22	magnolia_24	2026-02-09 18:00 UTC	1
02/10/26 00:22	magnolia_12	2026-02-09 18:00 UTC	1
02/10/26 01:22	magnolia_48	2026-02-09 18:00 UTC	1
02/10/26 01:22	magnolia_24	2026-02-09 18:00 UTC	1
02/10/26 01:23	magnolia_12	2026-02-09 18:00 UTC	1
02/10/26 02:21	magnolia_48	2026-02-10 02:00 UTC	1
02/10/26 02:22	magnolia_24	2026-02-10 02:00 UTC	1
02/10/26 02:22	magnolia_12	2026-02-10 02:00 UTC	1
02/10/26 03:21	magnolia_48	2026-02-09 18:00 UTC	1
02/10/26 03:22	magnolia_24	2026-02-09 18:00 UTC	1
02/10/26 03:22	magnolia_12	2026-02-09 18:00 UTC	1
02/10/26 04:21	magnolia_48	2026-02-09 18:00 UTC	1
02/10/26 04:22	magnolia_24	2026-02-09 18:00 UTC	1
02/10/26 04:22	magnolia_12	2026-02-09 18:00 UTC	1
02/10/26 05:22	magnolia_48	2026-02-09 18:00 UTC	1
02/10/26 05:22	magnolia_24	2026-02-09 18:00 UTC	1
02/10/26 05:22	magnolia_12	2026-02-09 18:00 UTC	1
02/10/26 06:22	magnolia_48	2026-02-09 18:00 UTC	1
02/10/26 06:22	magnolia_24	2026-02-09 18:00 UTC	1
02/10/26 06:22	magnolia_12	2026-02-09 18:00 UTC	1
02/10/26 07:21	magnolia_48	2026-02-09 18:00 UTC	1
02/10/26 07:22	magnolia_24	2026-02-09 18:00 UTC	1
02/10/26 07:22	magnolia_12	2026-02-09 18:00 UTC	1
02/10/26 08:21	magnolia_48	2026-02-09 18:00 UTC	1
02/10/26 08:22	magnolia_24	2026-02-09 18:00 UTC	1
02/10/26 08:22	magnolia_12	2026-02-09 18:00 UTC	1
02/10/26 09:21	magnolia_48	2026-02-09 18:00 UTC	1
02/10/26 09:22	magnolia_24	2026-02-09 18:00 UTC	1
02/10/26 09:22	magnolia_12	2026-02-09 18:00 UTC	1
02/10/26 10:21	magnolia_48	2026-02-09 18:00 UTC	1
02/10/26 10:22	magnolia_24	2026-02-09 18:00 UTC	1
02/10/26 10:22	magnolia_12	2026-02-09 18:00 UTC	1
02/10/26 11:21	magnolia_48	2026-02-09 18:00 UTC	1
02/10/26 11:22	magnolia_24	2026-02-09 18:00 UTC	1
02/10/26 11:22	magnolia_12	2026-02-09 18:00 UTC	1
02/10/26 12:21	magnolia_48	2026-02-09 18:00 UTC	1
02/10/26 12:22	magnolia_24	2026-02-09 18:00 UTC	1
02/10/26 12:22	magnolia_12	2026-02-09 18:00 UTC	1
02/10/26 13:21	magnolia_48	2026-02-09 18:00 UTC	1
02/10/26 13:22	magnolia_24	2026-02-09 18:00 UTC	1
02/10/26 13:22	magnolia_12	2026-02-09 18:00 UTC	1
02/10/26 14:21	magnolia_48	2026-02-09 18:00 UTC	1
02/10/26 14:22	magnolia_24	2026-02-09 18:00 UTC	1
02/10/26 14:22	magnolia_12	2026-02-09 18:00 UTC	1
02/10/26 15:21	magnolia_48	2026-02-09 18:00 UTC	1
02/10/26 15:22	magnolia_24	2026-02-09 18:00 UTC	1
02/10/26 15:22	magnolia_12	2026-02-09 18:00 UTC	1
02/10/26 16:21	magnolia_48	2026-02-09 18:00 UTC	1
02/10/26 16:22	magnolia_24	2026-02-09 18:00 UTC	1
02/10/26 16:22	magnolia_12	2026-02-09 18:00 UTC	1
02/10/26 17:22	magnolia_48	2026-02-09 18:00 UTC	1
02/10/26 17:22	magnolia_24	2026-02-09 18:00 UTC	1
02/10/26 17:22	magnolia_12	2026-02-09 18:00 UTC	1
02/10/26 18:20	Bird-Island_Water-Temperature_120hr	2026-02-09 18:00 UTC	1
02/10/26 18:20	MRE_Bird-Island_Water-Temperature_120hr	2026-02-09 18:00 UTC	1
02/10/26 18:20	Bird-Island_Water-Temperature_114hr	2026-02-09 18:00 UTC	1
02/10/26 18:20	MRE_Bird-Island_Water-Temperature_114hr	2026-02-09 18:00 UTC	1
02/10/26 18:20	Bird-Island_Water-Temperature_108hr	2026-02-09 18:00 UTC	1
02/10/26 18:20	MRE_Bird-Island_Water-Temperature_108hr	2026-02-09 18:00 UTC	1
02/10/26 18:20	Bird-Island_Water-Temperature_102hr	2026-02-09 18:00 UTC	1
02/10/26 18:20	MRE_Bird-Island_Water-Temperature_102hr	2026-02-09 18:00 UTC	1
02/10/26 18:20	Bird-Island_Water-Temperature_96hr	2026-02-09 18:00 UTC	1
02/10/26 18:20	MRE_Bird-Island_Water-Temperature_96hr	2026-02-09 18:00 UTC	1
02/10/26 18:20	Bird-Island_Water-Temperature_90hr	2026-02-09 18:00 UTC	1
02/10/26 18:20	MRE_Bird-Island_Water-Temperature_90hr	2026-02-09 18:00 UTC	1
02/10/26 18:20	Bird-Island_Water-Temperature_84hr	2026-02-09 18:00 UTC	1
02/10/26 18:20	MRE_Bird-Island_Water-Temperature_78hr	2026-02-09 18:00 UTC	1
02/10/26 18:20	MRE_Bird-Island_Water-Temperature_84hr	2026-02-09 18:00 UTC	1
02/10/26 18:20	Bird-Island_Water-Temperature_78hr	2026-02-09 18:00 UTC	1
02/10/26 18:20	Bird-Island_Water-Temperature_72hr	2026-02-09 18:00 UTC	1
02/10/26 18:20	MRE_Bird-Island_Water-Temperature_72hr	2026-02-09 18:00 UTC	1
02/10/26 18:20	Bird-Island_Water-Temperature_66hr	2026-02-09 18:00 UTC	1
02/10/26 18:20	MRE_Bird-Island_Water-Temperature_66hr	2026-02-09 18:00 UTC	1
02/10/26 18:20	MRE_Bird-Island_Water-Temperature_60hr	2026-02-09 18:00 UTC	1
02/10/26 18:20	Bird-Island_Water-Temperature_60hr	2026-02-09 18:00 UTC	1
02/10/26 18:20	Bird-Island_Water-Temperature_54hr	2026-02-09 18:00 UTC	1
02/10/26 18:20	MRE_Bird-Island_Water-Temperature_54hr	2026-02-09 18:00 UTC	1
02/10/26 18:21	Bird-Island_Water-Temperature_48hr	2026-02-09 18:00 UTC	1
02/10/26 18:21	MRE_Bird-Island_Water-Temperature_48hr	2026-02-09 18:00 UTC	1
02/10/26 18:21	magnolia_48	2026-02-09 18:00 UTC	1
02/10/26 18:21	Bird-Island_Water-Temperature_42hr	2026-02-09 18:00 UTC	1
02/10/26 18:21	MRE_Bird-Island_Water-Temperature_42hr	2026-02-09 18:00 UTC	1
02/10/26 18:21	MRE_Bird-Island_Water-Temperature_36hr	2026-02-09 18:00 UTC	1
02/10/26 18:21	Bird-Island_Water-Temperature_36hr	2026-02-09 18:00 UTC	1
02/10/26 18:21	MRE_Bird-Island_Water-Temperature_30hr	2026-02-09 18:00 UTC	1
02/10/26 18:21	Bird-Island_Water-Temperature_30hr	2026-02-09 18:00 UTC	1
02/10/26 18:21	Bird-Island_Water-Temperature_24hr	2026-02-09 18:00 UTC	1
02/10/26 18:21	MRE_Bird-Island_Water-Temperature_24hr	2026-02-09 18:00 UTC	1
02/10/26 18:21	magnolia_24	2026-02-09 18:00 UTC	1
02/10/26 18:21	MRE_Bird-Island_Water-Temperature_18hr	2026-02-09 18:00 UTC	1
02/10/26 18:21	Bird-Island_Water-Temperature_18hr	2026-02-09 18:00 UTC	1
02/10/26 18:21	Bird-Island_Water-Temperature_12hr	2026-02-09 18:00 UTC	1
02/10/26 18:21	MRE_Bird-Island_Water-Temperature_12hr	2026-02-09 18:00 UTC	1
02/10/26 18:21	magnolia_12	2026-02-09 18:00 UTC	1
02/10/26 18:21	Bird-Island_Water-Temperature_6hr	2026-02-09 18:00 UTC	1
02/10/26 18:21	MRE_Bird-Island_Water-Temperature_6hr	2026-02-09 18:00 UTC	1
02/10/26 18:21	Bird-Island_Water-Temperature_3hr	2026-02-09 18:00 UTC	1
02/10/26 18:21	MRE_Bird-Island_Water-Temperature_3hr	2026-02-09 18:00 UTC	1

The table shows 2026-02-09 18:00 UTC, and 2026-02-10 02:00 UTC missing for Magnolia Models and 2026-02-09 18:00 UTC missing for Bird Island Models. If I query that day on the light house stations that the models are dependent on I get the following results, with Port Lavaca and Port O’Connor having many more lines because we request more series from them.

South Bird Island

	Port Lavaca						Port O’Connor

2026-02-09 18:00 UTC — permanently missing, all three stations affected:

South Bird Island (013): both atp and wtp are NA — total sensor dropout
Port O'Connor (057): wsd and wdr are NA — wind sensors dropped out
Port Lavaca (033): wsd and wdr are NA — wind sensors dropped out

2026-02-10 02:00 UTC — partially missing:

Port O'Connor (057): still NA for wsd and wdr — wind sensors were still down
Port Lavaca (033) and South Bird Island (013): both fully present and valid

For the Bird Island models, the datapoint that we were missing in ingestion is very clearly still missing. Meaning that our issue with those models isn’t in semaphore ingestion but with Lighthouse.

Things are more complicated for the Magnolia models, I’m looking at Magnolia_12 specifically to simplify my investigation.

| [semaphore.svc@sherlock-dev magnolia_12]$ sed -n '3042,3735p' 2026_2_magnolia_12.log | grep -E "02/10/26|missing|Missing|DateRange|FAILED|series:|dWl|dWnDir|dWnSpd|dAirTmp|dWaterTmp|PortLavaca|PortOConnor" 02/10/26 00:22:35: DateRangeValidation: Series [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: [TimeDescription] -> fromDateTime: 2026-02-09 00:00:00+00:00, toDateTime: 2026-02-10 00:00:00+00:00. interval: 1:00:00, stalenessOffset: None is missing 1 values. 02/10/26 00:22:35: Missing time: 2026-02-09 18:00:00+00:00 02/10/26 00:22:35: Warning:: Prediction failed due to lack of data 02/10/26 00:22:35: Exception message: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/10/26 00:22:35: Full stack trace: raise Semaphore_Data_Exception(f'DateRangeValidation Failed in Data Gatherer! \n[Series] -> {series}') exceptions.Semaphore_Data_Exception: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/10/26 00:22:36: Model magnolia_12 FAILED - Null result inserted 02/10/26 00:22:36: Failed results: 02/10/26 00:22:36: dataValue dataUnit timeGenerated leadTime 02/10/26 01:23:03: DateRangeValidation: Series [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: [TimeDescription] -> fromDateTime: 2026-02-09 01:00:00+00:00, toDateTime: 2026-02-10 01:00:00+00:00. interval: 1:00:00, stalenessOffset: None is missing 1 values. 02/10/26 01:23:03: Missing time: 2026-02-09 18:00:00+00:00 02/10/26 01:23:03: Warning:: Prediction failed due to lack of data 02/10/26 01:23:03: Exception message: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/10/26 01:23:03: Full stack trace: raise Semaphore_Data_Exception(f'DateRangeValidation Failed in Data Gatherer! \n[Series] -> {series}') exceptions.Semaphore_Data_Exception: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/10/26 01:23:04: Model magnolia_12 FAILED - Null result inserted 02/10/26 01:23:04: Failed results: 02/10/26 01:23:04: dataValue dataUnit timeGenerated leadTime 02/10/26 02:22:29: DateRangeValidation: Series [SeriesDescription] -> source: LIGHTHOUSE, series: dWl, location: PortOConnor, datum: MLLW, dataIntegrityDescription: None, verificationOverride: None, timeDescription: [TimeDescription] -> fromDateTime: 2026-02-09 02:00:00+00:00, toDateTime: 2026-02-10 02:00:00+00:00. interval: 1:00:00, stalenessOffset: None is missing 1 values. 02/10/26 02:22:29: Missing time: 2026-02-10 02:00:00+00:00 02/10/26 02:22:29: Warning:: Prediction failed due to lack of data 02/10/26 02:22:29: Exception message: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWl, location: PortOConnor, datum: MLLW, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/10/26 02:22:29: Full stack trace: raise Semaphore_Data_Exception(f'DateRangeValidation Failed in Data Gatherer! \n[Series] -> {series}') exceptions.Semaphore_Data_Exception: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWl, location: PortOConnor, datum: MLLW, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/10/26 02:22:29: Model magnolia_12 FAILED - Null result inserted 02/10/26 02:22:29: Failed results: 02/10/26 02:22:29: dataValue dataUnit timeGenerated leadTime 02/10/26 03:22:33: DateRangeValidation: Series [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: [TimeDescription] -> fromDateTime: 2026-02-09 03:00:00+00:00, toDateTime: 2026-02-10 03:00:00+00:00. interval: 1:00:00, stalenessOffset: None is missing 1 values. 02/10/26 03:22:33: Missing time: 2026-02-09 18:00:00+00:00 02/10/26 03:22:33: Warning:: Prediction failed due to lack of data 02/10/26 03:22:33: Exception message: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/10/26 03:22:33: Full stack trace: raise Semaphore_Data_Exception(f'DateRangeValidation Failed in Data Gatherer! \n[Series] -> {series}') exceptions.Semaphore_Data_Exception: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/10/26 03:22:33: Model magnolia_12 FAILED - Null result inserted 02/10/26 03:22:33: Failed results: 02/10/26 03:22:33: dataValue dataUnit timeGenerated leadTime 02/10/26 04:22:31: DateRangeValidation: Series [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: [TimeDescription] -> fromDateTime: 2026-02-09 04:00:00+00:00, toDateTime: 2026-02-10 04:00:00+00:00. interval: 1:00:00, stalenessOffset: None is missing 1 values. 02/10/26 04:22:31: Missing time: 2026-02-09 18:00:00+00:00 02/10/26 04:22:31: Warning:: Prediction failed due to lack of data 02/10/26 04:22:31: Exception message: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/10/26 04:22:31: Full stack trace: raise Semaphore_Data_Exception(f'DateRangeValidation Failed in Data Gatherer! \n[Series] -> {series}') exceptions.Semaphore_Data_Exception: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/10/26 04:22:31: Model magnolia_12 FAILED - Null result inserted 02/10/26 04:22:31: Failed results: 02/10/26 04:22:31: dataValue dataUnit timeGenerated leadTime 02/10/26 05:22:35: DateRangeValidation: Series [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: [TimeDescription] -> fromDateTime: 2026-02-09 05:00:00+00:00, toDateTime: 2026-02-10 05:00:00+00:00. interval: 1:00:00, stalenessOffset: None is missing 1 values. 02/10/26 05:22:35: Missing time: 2026-02-09 18:00:00+00:00 02/10/26 05:22:35: Warning:: Prediction failed due to lack of data 02/10/26 05:22:35: Exception message: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/10/26 05:22:35: Full stack trace: raise Semaphore_Data_Exception(f'DateRangeValidation Failed in Data Gatherer! \n[Series] -> {series}') exceptions.Semaphore_Data_Exception: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/10/26 05:22:35: Model magnolia_12 FAILED - Null result inserted 02/10/26 05:22:35: Failed results: 02/10/26 05:22:35: dataValue dataUnit timeGenerated leadTime 02/10/26 06:22:37: DateRangeValidation: Series [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: [TimeDescription] -> fromDateTime: 2026-02-09 06:00:00+00:00, toDateTime: 2026-02-10 06:00:00+00:00. interval: 1:00:00, stalenessOffset: None is missing 1 values. 02/10/26 06:22:37: Missing time: 2026-02-09 18:00:00+00:00 02/10/26 06:22:37: Warning:: Prediction failed due to lack of data 02/10/26 06:22:37: Exception message: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/10/26 06:22:37: Full stack trace: raise Semaphore_Data_Exception(f'DateRangeValidation Failed in Data Gatherer! \n[Series] -> {series}') exceptions.Semaphore_Data_Exception: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/10/26 06:22:39: Model magnolia_12 FAILED - Null result inserted 02/10/26 06:22:39: Failed results: 02/10/26 06:22:39: dataValue dataUnit timeGenerated leadTime 02/10/26 07:22:27: DateRangeValidation: Series [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: [TimeDescription] -> fromDateTime: 2026-02-09 07:00:00+00:00, toDateTime: 2026-02-10 07:00:00+00:00. interval: 1:00:00, stalenessOffset: None is missing 1 values. 02/10/26 07:22:27: Missing time: 2026-02-09 18:00:00+00:00 02/10/26 07:22:27: Warning:: Prediction failed due to lack of data 02/10/26 07:22:27: Exception message: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/10/26 07:22:27: Full stack trace: raise Semaphore_Data_Exception(f'DateRangeValidation Failed in Data Gatherer! \n[Series] -> {series}') exceptions.Semaphore_Data_Exception: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/10/26 07:22:27: Model magnolia_12 FAILED - Null result inserted 02/10/26 07:22:27: Failed results: 02/10/26 07:22:27: dataValue dataUnit timeGenerated leadTime 02/10/26 08:22:29: DateRangeValidation: Series [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: [TimeDescription] -> fromDateTime: 2026-02-09 08:00:00+00:00, toDateTime: 2026-02-10 08:00:00+00:00. interval: 1:00:00, stalenessOffset: None is missing 1 values. 02/10/26 08:22:29: Missing time: 2026-02-09 18:00:00+00:00 02/10/26 08:22:29: Warning:: Prediction failed due to lack of data 02/10/26 08:22:29: Exception message: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/10/26 08:22:29: Full stack trace: raise Semaphore_Data_Exception(f'DateRangeValidation Failed in Data Gatherer! \n[Series] -> {series}') exceptions.Semaphore_Data_Exception: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/10/26 08:22:30: Model magnolia_12 FAILED - Null result inserted 02/10/26 08:22:30: Failed results: 02/10/26 08:22:30: dataValue dataUnit timeGenerated leadTime 02/10/26 09:22:33: DateRangeValidation: Series [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: [TimeDescription] -> fromDateTime: 2026-02-09 09:00:00+00:00, toDateTime: 2026-02-10 09:00:00+00:00. interval: 1:00:00, stalenessOffset: None is missing 1 values. 02/10/26 09:22:33: Missing time: 2026-02-09 18:00:00+00:00 02/10/26 09:22:33: Warning:: Prediction failed due to lack of data 02/10/26 09:22:33: Exception message: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/10/26 09:22:33: Full stack trace: raise Semaphore_Data_Exception(f'DateRangeValidation Failed in Data Gatherer! \n[Series] -> {series}') exceptions.Semaphore_Data_Exception: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/10/26 09:22:33: Model magnolia_12 FAILED - Null result inserted 02/10/26 09:22:33: Failed results: 02/10/26 09:22:33: dataValue dataUnit timeGenerated leadTime 02/10/26 10:22:32: DateRangeValidation: Series [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: [TimeDescription] -> fromDateTime: 2026-02-09 10:00:00+00:00, toDateTime: 2026-02-10 10:00:00+00:00. interval: 1:00:00, stalenessOffset: None is missing 1 values. 02/10/26 10:22:32: Missing time: 2026-02-09 18:00:00+00:00 02/10/26 10:22:32: Warning:: Prediction failed due to lack of data 02/10/26 10:22:32: Exception message: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/10/26 10:22:32: Full stack trace: raise Semaphore_Data_Exception(f'DateRangeValidation Failed in Data Gatherer! \n[Series] -> {series}') exceptions.Semaphore_Data_Exception: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/10/26 10:22:32: Model magnolia_12 FAILED - Null result inserted 02/10/26 10:22:32: Failed results: 02/10/26 10:22:32: dataValue dataUnit timeGenerated leadTime 02/10/26 11:22:29: DateRangeValidation: Series [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: [TimeDescription] -> fromDateTime: 2026-02-09 11:00:00+00:00, toDateTime: 2026-02-10 11:00:00+00:00. interval: 1:00:00, stalenessOffset: None is missing 1 values. 02/10/26 11:22:29: Missing time: 2026-02-09 18:00:00+00:00 02/10/26 11:22:29: Warning:: Prediction failed due to lack of data 02/10/26 11:22:29: Exception message: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/10/26 11:22:29: Full stack trace: raise Semaphore_Data_Exception(f'DateRangeValidation Failed in Data Gatherer! \n[Series] -> {series}') exceptions.Semaphore_Data_Exception: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/10/26 11:22:30: Model magnolia_12 FAILED - Null result inserted 02/10/26 11:22:30: Failed results: 02/10/26 11:22:30: dataValue dataUnit timeGenerated leadTime 02/10/26 12:22:30: DateRangeValidation: Series [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: [TimeDescription] -> fromDateTime: 2026-02-09 12:00:00+00:00, toDateTime: 2026-02-10 12:00:00+00:00. interval: 1:00:00, stalenessOffset: None is missing 1 values. 02/10/26 12:22:30: Missing time: 2026-02-09 18:00:00+00:00 02/10/26 12:22:30: Warning:: Prediction failed due to lack of data 02/10/26 12:22:30: Exception message: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/10/26 12:22:30: Full stack trace: raise Semaphore_Data_Exception(f'DateRangeValidation Failed in Data Gatherer! \n[Series] -> {series}') exceptions.Semaphore_Data_Exception: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/10/26 12:22:30: Model magnolia_12 FAILED - Null result inserted 02/10/26 12:22:30: Failed results: 02/10/26 12:22:30: dataValue dataUnit timeGenerated leadTime 02/10/26 13:22:33: DateRangeValidation: Series [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: [TimeDescription] -> fromDateTime: 2026-02-09 13:00:00+00:00, toDateTime: 2026-02-10 13:00:00+00:00. interval: 1:00:00, stalenessOffset: None is missing 1 values. 02/10/26 13:22:33: Missing time: 2026-02-09 18:00:00+00:00 02/10/26 13:22:33: Warning:: Prediction failed due to lack of data 02/10/26 13:22:33: Exception message: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/10/26 13:22:33: Full stack trace: raise Semaphore_Data_Exception(f'DateRangeValidation Failed in Data Gatherer! \n[Series] -> {series}') exceptions.Semaphore_Data_Exception: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/10/26 13:22:33: Model magnolia_12 FAILED - Null result inserted 02/10/26 13:22:33: Failed results: 02/10/26 13:22:33: dataValue dataUnit timeGenerated leadTime 02/10/26 14:22:31: DateRangeValidation: Series [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: [TimeDescription] -> fromDateTime: 2026-02-09 14:00:00+00:00, toDateTime: 2026-02-10 14:00:00+00:00. interval: 1:00:00, stalenessOffset: None is missing 1 values. 02/10/26 14:22:31: Missing time: 2026-02-09 18:00:00+00:00 02/10/26 14:22:31: Warning:: Prediction failed due to lack of data 02/10/26 14:22:31: Exception message: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/10/26 14:22:31: Full stack trace: raise Semaphore_Data_Exception(f'DateRangeValidation Failed in Data Gatherer! \n[Series] -> {series}') exceptions.Semaphore_Data_Exception: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/10/26 14:22:31: Model magnolia_12 FAILED - Null result inserted 02/10/26 14:22:31: Failed results: 02/10/26 14:22:31: dataValue dataUnit timeGenerated leadTime 02/10/26 15:22:26: DateRangeValidation: Series [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: [TimeDescription] -> fromDateTime: 2026-02-09 15:00:00+00:00, toDateTime: 2026-02-10 15:00:00+00:00. interval: 1:00:00, stalenessOffset: None is missing 1 values. 02/10/26 15:22:26: Missing time: 2026-02-09 18:00:00+00:00 02/10/26 15:22:26: Warning:: Prediction failed due to lack of data 02/10/26 15:22:26: Exception message: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/10/26 15:22:26: Full stack trace: raise Semaphore_Data_Exception(f'DateRangeValidation Failed in Data Gatherer! \n[Series] -> {series}') exceptions.Semaphore_Data_Exception: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/10/26 15:22:26: Model magnolia_12 FAILED - Null result inserted 02/10/26 15:22:26: Failed results: 02/10/26 15:22:26: dataValue dataUnit timeGenerated leadTime 02/10/26 16:22:33: DateRangeValidation: Series [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: [TimeDescription] -> fromDateTime: 2026-02-09 16:00:00+00:00, toDateTime: 2026-02-10 16:00:00+00:00. interval: 1:00:00, stalenessOffset: None is missing 1 values. 02/10/26 16:22:33: Missing time: 2026-02-09 18:00:00+00:00 02/10/26 16:22:33: Warning:: Prediction failed due to lack of data 02/10/26 16:22:33: Exception message: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/10/26 16:22:33: Full stack trace: raise Semaphore_Data_Exception(f'DateRangeValidation Failed in Data Gatherer! \n[Series] -> {series}') exceptions.Semaphore_Data_Exception: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/10/26 16:22:33: Model magnolia_12 FAILED - Null result inserted 02/10/26 16:22:33: Failed results: 02/10/26 16:22:33: dataValue dataUnit timeGenerated leadTime 02/10/26 17:22:47: DateRangeValidation: Series [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: [TimeDescription] -> fromDateTime: 2026-02-09 17:00:00+00:00, toDateTime: 2026-02-10 17:00:00+00:00. interval: 1:00:00, stalenessOffset: None is missing 1 values. 02/10/26 17:22:47: Missing time: 2026-02-09 18:00:00+00:00 02/10/26 17:22:47: Warning:: Prediction failed due to lack of data 02/10/26 17:22:47: Exception message: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/10/26 17:22:47: Full stack trace: raise Semaphore_Data_Exception(f'DateRangeValidation Failed in Data Gatherer! \n[Series] -> {series}') exceptions.Semaphore_Data_Exception: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/10/26 17:22:47: Model magnolia_12 FAILED - Null result inserted 02/10/26 17:22:47: Failed results: 02/10/26 17:22:47: dataValue dataUnit timeGenerated leadTime 02/10/26 18:21:41: DateRangeValidation: Series [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: [TimeDescription] -> fromDateTime: 2026-02-09 18:00:00+00:00, toDateTime: 2026-02-10 18:00:00+00:00. interval: 1:00:00, stalenessOffset: None is missing 1 values. 02/10/26 18:21:41: Missing time: 2026-02-09 18:00:00+00:00 02/10/26 18:21:41: Warning:: Prediction failed due to lack of data 02/10/26 18:21:41: Exception message: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/10/26 18:21:41: Full stack trace: raise Semaphore_Data_Exception(f'DateRangeValidation Failed in Data Gatherer! \n[Series] -> {series}') exceptions.Semaphore_Data_Exception: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/10/26 18:21:41: Model magnolia_12 FAILED - Null result inserted 02/10/26 18:21:41: Failed results: 02/10/26 18:21:41: dataValue dataUnit timeGenerated leadTime 02/10/26 19:22:54: Model magnolia_12 completed successfully ✓ 02/10/26 19:22:54: Results inserted: 02/10/26 19:22:54: dataValue dataUnit timeGenerated leadTime 02/10/26 20:23:05: Model magnolia_12 completed successfully ✓ 02/10/26 20:23:05: Results inserted: 02/10/26 20:23:05: dataValue dataUnit timeGenerated leadTime 02/10/26 21:22:56: Model magnolia_12 completed successfully ✓ 02/10/26 21:22:56: Results inserted: 02/10/26 21:22:56: dataValue dataUnit timeGenerated leadTime 02/10/26 22:22:42: Model magnolia_12 completed successfully ✓ 02/10/26 22:22:42: Results inserted: 02/10/26 22:22:42: dataValue dataUnit timeGenerated leadTime 02/10/26 23:22:40: Model magnolia_12 completed successfully ✓ 02/10/26 23:22:40: Results inserted: 02/10/26 23:22:40: dataValue dataUnit timeGenerated leadTime | | :---- |

20 out of 24 magnolia_12 runs on February 10th failed due to a single missing wind direction observation at Port Lavaca at 18:00 UTC on February 9th, which was permanently lost at the sensor level and never populated in LIGHTHOUSE. One additional run failed due to a suspected latency issue on a separate series.

Every failure from 00:22 through 18:21 (19 consecutive runs) failed on the exact same thing: dWnDir at PortLavaca, missing 2026-02-09 18:00 UTC. That's the wind direction sensor dropout we confirmed is permanently NA in LIGHTHOUSE — nothing Semaphore can do about it. The model kept failing on the same point all day because that timestamp stayed inside the sliding 24-hour lookback window until the 18:00 CST run finally slid past it.

The 02:22 run is the odd one out — it failed on dWl at Port O’Connor missing 2026-02-10 02:00 UTC. I think that this is a latency case that we see only with the Magnolia Models, specifically with Port O’Connor station. The data was published to LIGHTHOUSE later (since I can see it in the CSV that I downloaded) and is valid now, but wasn’t available at the time of request. To look into the latency issue further I chose a day where only Magnolia models had issues:

Monday, February 16
02/16/26 00:22	magnolia_48	2026-02-16 00:00 UTC	1
02/16/26 00:22	magnolia_24	2026-02-16 00:00 UTC	1
02/16/26 00:22	magnolia_12	2026-02-16 00:00 UTC	1
02/16/26 01:22	magnolia_48	2026-02-15 18:00 UTC	1
02/16/26 01:22	magnolia_24	2026-02-15 18:00 UTC	1
02/16/26 01:22	magnolia_12	2026-02-15 18:00 UTC	1
02/16/26 02:22	magnolia_48	2026-02-15 18:00 UTC	1
02/16/26 02:22	magnolia_24	2026-02-15 18:00 UTC	1
02/16/26 02:23	magnolia_12	2026-02-15 18:00 UTC	1
02/16/26 03:22	magnolia_48	2026-02-16 03:00 UTC	1
02/16/26 03:22	magnolia_24	2026-02-16 03:00 UTC	1
02/16/26 03:22	magnolia_12	2026-02-16 03:00 UTC	1
02/16/26 04:22	magnolia_48	2026-02-15 18:00 UTC	1
02/16/26 04:22	magnolia_24	2026-02-15 18:00 UTC	1
02/16/26 04:23	magnolia_12	2026-02-15 18:00 UTC	1
02/16/26 05:22	magnolia_48	2026-02-15 18:00 UTC	1
02/16/26 05:22	magnolia_24	2026-02-15 18:00 UTC	1
02/16/26 05:22	magnolia_12	2026-02-15 18:00 UTC	1
02/16/26 06:22	magnolia_48	2026-02-15 18:00 UTC	1
02/16/26 06:23	magnolia_24	2026-02-15 18:00 UTC	1
02/16/26 06:23	magnolia_12	2026-02-15 18:00 UTC	1
02/16/26 07:22	magnolia_48	2026-02-15 18:00 UTC	1
02/16/26 07:22	magnolia_24	2026-02-15 18:00 UTC	1
02/16/26 07:22	magnolia_12	2026-02-15 18:00 UTC	1
02/16/26 08:22	magnolia_48	2026-02-15 18:00 UTC	1
02/16/26 08:22	magnolia_24	2026-02-15 18:00 UTC	1
02/16/26 08:22	magnolia_12	2026-02-15 18:00 UTC	1
02/16/26 09:22	magnolia_48	2026-02-15 18:00 UTC	1
02/16/26 09:22	magnolia_24	2026-02-15 18:00 UTC	1
02/16/26 09:23	magnolia_12	2026-02-15 18:00 UTC	1
02/16/26 10:22	magnolia_48	2026-02-15 18:00 UTC	1
02/16/26 10:22	magnolia_24	2026-02-15 18:00 UTC	1
02/16/26 10:23	magnolia_12	2026-02-15 18:00 UTC	1
02/16/26 11:22	magnolia_48	2026-02-15 18:00 UTC	1
02/16/26 11:22	magnolia_24	2026-02-15 18:00 UTC	1
02/16/26 11:23	magnolia_12	2026-02-15 18:00 UTC	1
02/16/26 12:22	magnolia_48	2026-02-15 18:00 UTC	1
02/16/26 12:22	magnolia_24	2026-02-15 18:00 UTC	1
02/16/26 12:23	magnolia_12	2026-02-15 18:00 UTC	1
02/16/26 13:22	magnolia_48	2026-02-15 18:00 UTC	1
02/16/26 13:22	magnolia_24	2026-02-15 18:00 UTC	1
02/16/26 13:22	magnolia_12	2026-02-15 18:00 UTC	1
02/16/26 14:22	magnolia_48	2026-02-15 18:00 UTC	1
02/16/26 14:22	magnolia_24	2026-02-15 18:00 UTC	1
02/16/26 14:22	magnolia_12	2026-02-15 18:00 UTC	1
02/16/26 15:22	magnolia_48	2026-02-15 18:00 UTC	1
02/16/26 15:22	magnolia_24	2026-02-15 18:00 UTC	1
02/16/26 15:22	magnolia_12	2026-02-15 18:00 UTC	1
02/16/26 16:22	magnolia_48	2026-02-15 18:00 UTC	1
02/16/26 16:22	magnolia_24	2026-02-15 18:00 UTC	1
02/16/26 16:22	magnolia_12	2026-02-15 18:00 UTC	1
02/16/26 17:22	magnolia_48	2026-02-15 18:00 UTC	1
02/16/26 17:22	magnolia_24	2026-02-15 18:00 UTC	1
02/16/26 17:23	magnolia_12	2026-02-15 18:00 UTC	1
02/16/26 18:22	magnolia_48	2026-02-15 18:00 UTC	1
02/16/26 18:22	magnolia_24	2026-02-15 18:00 UTC	1
02/16/26 18:22	magnolia_12	2026-02-15 18:00 UTC	1

| [semaphore.svc@sherlock-dev magnolia_12]$ sed -n '6613,7306p' 2026_2_magnolia_12.log | grep -E "02/16/26|missing|Missing|DateRange|FAILED|series:|dWl|dWnDir|dWnSpd|dAirTmp|dWaterTmp|PortLavaca|PortOConnor" 02/16/26 00:22:56: DateRangeValidation: Series [SeriesDescription] -> source: LIGHTHOUSE, series: dWl, location: PortOConnor, datum: MLLW, dataIntegrityDescription: None, verificationOverride: None, timeDescription: [TimeDescription] -> fromDateTime: 2026-02-15 00:00:00+00:00, toDateTime: 2026-02-16 00:00:00+00:00. interval: 1:00:00, stalenessOffset: None is missing 1 values. 02/16/26 00:22:56: Missing time: 2026-02-16 00:00:00+00:00 02/16/26 00:22:56: Warning:: Prediction failed due to lack of data 02/16/26 00:22:56: Exception message: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWl, location: PortOConnor, datum: MLLW, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/16/26 00:22:56: Full stack trace: raise Semaphore_Data_Exception(f'DateRangeValidation Failed in Data Gatherer! \n[Series] -> {series}') exceptions.Semaphore_Data_Exception: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWl, location: PortOConnor, datum: MLLW, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/16/26 00:22:56: Model magnolia_12 FAILED - Null result inserted 02/16/26 00:22:56: Failed results: 02/16/26 00:22:56: dataValue dataUnit timeGenerated leadTime 02/16/26 01:22:47: DateRangeValidation: Series [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: [TimeDescription] -> fromDateTime: 2026-02-15 01:00:00+00:00, toDateTime: 2026-02-16 01:00:00+00:00. interval: 1:00:00, stalenessOffset: None is missing 1 values. 02/16/26 01:22:47: Missing time: 2026-02-15 18:00:00+00:00 02/16/26 01:22:47: Warning:: Prediction failed due to lack of data 02/16/26 01:22:47: Exception message: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/16/26 01:22:47: Full stack trace: raise Semaphore_Data_Exception(f'DateRangeValidation Failed in Data Gatherer! \n[Series] -> {series}') exceptions.Semaphore_Data_Exception: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/16/26 01:22:48: Model magnolia_12 FAILED - Null result inserted 02/16/26 01:22:48: Failed results: 02/16/26 01:22:48: dataValue dataUnit timeGenerated leadTime 02/16/26 02:23:00: DateRangeValidation: Series [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: [TimeDescription] -> fromDateTime: 2026-02-15 02:00:00+00:00, toDateTime: 2026-02-16 02:00:00+00:00. interval: 1:00:00, stalenessOffset: None is missing 1 values. 02/16/26 02:23:01: Missing time: 2026-02-15 18:00:00+00:00 02/16/26 02:23:01: Warning:: Prediction failed due to lack of data 02/16/26 02:23:01: Exception message: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/16/26 02:23:01: Full stack trace: raise Semaphore_Data_Exception(f'DateRangeValidation Failed in Data Gatherer! \n[Series] -> {series}') exceptions.Semaphore_Data_Exception: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/16/26 02:23:01: Model magnolia_12 FAILED - Null result inserted 02/16/26 02:23:01: Failed results: 02/16/26 02:23:01: dataValue dataUnit timeGenerated leadTime 02/16/26 03:22:48: DateRangeValidation: Series [SeriesDescription] -> source: LIGHTHOUSE, series: dWl, location: PortOConnor, datum: MLLW, dataIntegrityDescription: None, verificationOverride: None, timeDescription: [TimeDescription] -> fromDateTime: 2026-02-15 03:00:00+00:00, toDateTime: 2026-02-16 03:00:00+00:00. interval: 1:00:00, stalenessOffset: None is missing 1 values. 02/16/26 03:22:48: Missing time: 2026-02-16 03:00:00+00:00 02/16/26 03:22:48: Warning:: Prediction failed due to lack of data 02/16/26 03:22:48: Exception message: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWl, location: PortOConnor, datum: MLLW, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/16/26 03:22:48: Full stack trace: raise Semaphore_Data_Exception(f'DateRangeValidation Failed in Data Gatherer! \n[Series] -> {series}') exceptions.Semaphore_Data_Exception: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWl, location: PortOConnor, datum: MLLW, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/16/26 03:22:48: Model magnolia_12 FAILED - Null result inserted 02/16/26 03:22:48: Failed results: 02/16/26 03:22:48: dataValue dataUnit timeGenerated leadTime 02/16/26 04:23:06: DateRangeValidation: Series [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: [TimeDescription] -> fromDateTime: 2026-02-15 04:00:00+00:00, toDateTime: 2026-02-16 04:00:00+00:00. interval: 1:00:00, stalenessOffset: None is missing 1 values. 02/16/26 04:23:06: Missing time: 2026-02-15 18:00:00+00:00 02/16/26 04:23:06: Warning:: Prediction failed due to lack of data 02/16/26 04:23:06: Exception message: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/16/26 04:23:06: Full stack trace: raise Semaphore_Data_Exception(f'DateRangeValidation Failed in Data Gatherer! \n[Series] -> {series}') exceptions.Semaphore_Data_Exception: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/16/26 04:23:06: Model magnolia_12 FAILED - Null result inserted 02/16/26 04:23:06: Failed results: 02/16/26 04:23:06: dataValue dataUnit timeGenerated leadTime 02/16/26 05:22:59: DateRangeValidation: Series [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: [TimeDescription] -> fromDateTime: 2026-02-15 05:00:00+00:00, toDateTime: 2026-02-16 05:00:00+00:00. interval: 1:00:00, stalenessOffset: None is missing 1 values. 02/16/26 05:22:59: Missing time: 2026-02-15 18:00:00+00:00 02/16/26 05:22:59: Warning:: Prediction failed due to lack of data 02/16/26 05:22:59: Exception message: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/16/26 05:22:59: Full stack trace: raise Semaphore_Data_Exception(f'DateRangeValidation Failed in Data Gatherer! \n[Series] -> {series}') exceptions.Semaphore_Data_Exception: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/16/26 05:22:59: Model magnolia_12 FAILED - Null result inserted 02/16/26 05:22:59: Failed results: 02/16/26 05:22:59: dataValue dataUnit timeGenerated leadTime 02/16/26 06:23:31: DateRangeValidation: Series [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: [TimeDescription] -> fromDateTime: 2026-02-15 06:00:00+00:00, toDateTime: 2026-02-16 06:00:00+00:00. interval: 1:00:00, stalenessOffset: None is missing 1 values. 02/16/26 06:23:31: Missing time: 2026-02-15 18:00:00+00:00 02/16/26 06:23:31: Warning:: Prediction failed due to lack of data 02/16/26 06:23:31: Exception message: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/16/26 06:23:31: Full stack trace: raise Semaphore_Data_Exception(f'DateRangeValidation Failed in Data Gatherer! \n[Series] -> {series}') exceptions.Semaphore_Data_Exception: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/16/26 06:23:31: Model magnolia_12 FAILED - Null result inserted 02/16/26 06:23:31: Failed results: 02/16/26 06:23:31: dataValue dataUnit timeGenerated leadTime 02/16/26 07:22:57: DateRangeValidation: Series [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: [TimeDescription] -> fromDateTime: 2026-02-15 07:00:00+00:00, toDateTime: 2026-02-16 07:00:00+00:00. interval: 1:00:00, stalenessOffset: None is missing 1 values. 02/16/26 07:22:57: Missing time: 2026-02-15 18:00:00+00:00 02/16/26 07:22:57: Warning:: Prediction failed due to lack of data 02/16/26 07:22:57: Exception message: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/16/26 07:22:57: Full stack trace: raise Semaphore_Data_Exception(f'DateRangeValidation Failed in Data Gatherer! \n[Series] -> {series}') exceptions.Semaphore_Data_Exception: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/16/26 07:22:58: Model magnolia_12 FAILED - Null result inserted 02/16/26 07:22:58: Failed results: 02/16/26 07:22:58: dataValue dataUnit timeGenerated leadTime 02/16/26 08:22:53: DateRangeValidation: Series [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: [TimeDescription] -> fromDateTime: 2026-02-15 08:00:00+00:00, toDateTime: 2026-02-16 08:00:00+00:00. interval: 1:00:00, stalenessOffset: None is missing 1 values. 02/16/26 08:22:53: Missing time: 2026-02-15 18:00:00+00:00 02/16/26 08:22:53: Warning:: Prediction failed due to lack of data 02/16/26 08:22:53: Exception message: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/16/26 08:22:53: Full stack trace: raise Semaphore_Data_Exception(f'DateRangeValidation Failed in Data Gatherer! \n[Series] -> {series}') exceptions.Semaphore_Data_Exception: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/16/26 08:22:54: Model magnolia_12 FAILED - Null result inserted 02/16/26 08:22:54: Failed results: 02/16/26 08:22:54: dataValue dataUnit timeGenerated leadTime 02/16/26 09:23:05: DateRangeValidation: Series [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: [TimeDescription] -> fromDateTime: 2026-02-15 09:00:00+00:00, toDateTime: 2026-02-16 09:00:00+00:00. interval: 1:00:00, stalenessOffset: None is missing 1 values. 02/16/26 09:23:05: Missing time: 2026-02-15 18:00:00+00:00 02/16/26 09:23:05: Warning:: Prediction failed due to lack of data 02/16/26 09:23:05: Exception message: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/16/26 09:23:05: Full stack trace: raise Semaphore_Data_Exception(f'DateRangeValidation Failed in Data Gatherer! \n[Series] -> {series}') exceptions.Semaphore_Data_Exception: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/16/26 09:23:05: Model magnolia_12 FAILED - Null result inserted 02/16/26 09:23:05: Failed results: 02/16/26 09:23:05: dataValue dataUnit timeGenerated leadTime 02/16/26 10:23:03: DateRangeValidation: Series [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: [TimeDescription] -> fromDateTime: 2026-02-15 10:00:00+00:00, toDateTime: 2026-02-16 10:00:00+00:00. interval: 1:00:00, stalenessOffset: None is missing 1 values. 02/16/26 10:23:03: Missing time: 2026-02-15 18:00:00+00:00 02/16/26 10:23:03: Warning:: Prediction failed due to lack of data 02/16/26 10:23:03: Exception message: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/16/26 10:23:03: Full stack trace: raise Semaphore_Data_Exception(f'DateRangeValidation Failed in Data Gatherer! \n[Series] -> {series}') exceptions.Semaphore_Data_Exception: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/16/26 10:23:03: Model magnolia_12 FAILED - Null result inserted 02/16/26 10:23:03: Failed results: 02/16/26 10:23:03: dataValue dataUnit timeGenerated leadTime 02/16/26 11:23:05: DateRangeValidation: Series [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: [TimeDescription] -> fromDateTime: 2026-02-15 11:00:00+00:00, toDateTime: 2026-02-16 11:00:00+00:00. interval: 1:00:00, stalenessOffset: None is missing 1 values. 02/16/26 11:23:05: Missing time: 2026-02-15 18:00:00+00:00 02/16/26 11:23:05: Warning:: Prediction failed due to lack of data 02/16/26 11:23:05: Exception message: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/16/26 11:23:05: Full stack trace: raise Semaphore_Data_Exception(f'DateRangeValidation Failed in Data Gatherer! \n[Series] -> {series}') exceptions.Semaphore_Data_Exception: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/16/26 11:23:05: Model magnolia_12 FAILED - Null result inserted 02/16/26 11:23:05: Failed results: 02/16/26 11:23:05: dataValue dataUnit timeGenerated leadTime 02/16/26 12:23:13: DateRangeValidation: Series [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: [TimeDescription] -> fromDateTime: 2026-02-15 12:00:00+00:00, toDateTime: 2026-02-16 12:00:00+00:00. interval: 1:00:00, stalenessOffset: None is missing 1 values. 02/16/26 12:23:13: Missing time: 2026-02-15 18:00:00+00:00 02/16/26 12:23:13: Warning:: Prediction failed due to lack of data 02/16/26 12:23:13: Exception message: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/16/26 12:23:13: Full stack trace: raise Semaphore_Data_Exception(f'DateRangeValidation Failed in Data Gatherer! \n[Series] -> {series}') exceptions.Semaphore_Data_Exception: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/16/26 12:23:13: Model magnolia_12 FAILED - Null result inserted 02/16/26 12:23:13: Failed results: 02/16/26 12:23:13: dataValue dataUnit timeGenerated leadTime 02/16/26 13:22:53: DateRangeValidation: Series [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: [TimeDescription] -> fromDateTime: 2026-02-15 13:00:00+00:00, toDateTime: 2026-02-16 13:00:00+00:00. interval: 1:00:00, stalenessOffset: None is missing 1 values. 02/16/26 13:22:53: Missing time: 2026-02-15 18:00:00+00:00 02/16/26 13:22:53: Warning:: Prediction failed due to lack of data 02/16/26 13:22:53: Exception message: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/16/26 13:22:53: Full stack trace: raise Semaphore_Data_Exception(f'DateRangeValidation Failed in Data Gatherer! \n[Series] -> {series}') exceptions.Semaphore_Data_Exception: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/16/26 13:22:53: Model magnolia_12 FAILED - Null result inserted 02/16/26 13:22:53: Failed results: 02/16/26 13:22:53: dataValue dataUnit timeGenerated leadTime 02/16/26 14:22:55: DateRangeValidation: Series [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: [TimeDescription] -> fromDateTime: 2026-02-15 14:00:00+00:00, toDateTime: 2026-02-16 14:00:00+00:00. interval: 1:00:00, stalenessOffset: None is missing 1 values. 02/16/26 14:22:55: Missing time: 2026-02-15 18:00:00+00:00 02/16/26 14:22:55: Warning:: Prediction failed due to lack of data 02/16/26 14:22:55: Exception message: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/16/26 14:22:55: Full stack trace: raise Semaphore_Data_Exception(f'DateRangeValidation Failed in Data Gatherer! \n[Series] -> {series}') exceptions.Semaphore_Data_Exception: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/16/26 14:22:55: Model magnolia_12 FAILED - Null result inserted 02/16/26 14:22:55: Failed results: 02/16/26 14:22:55: dataValue dataUnit timeGenerated leadTime 02/16/26 15:22:55: DateRangeValidation: Series [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: [TimeDescription] -> fromDateTime: 2026-02-15 15:00:00+00:00, toDateTime: 2026-02-16 15:00:00+00:00. interval: 1:00:00, stalenessOffset: None is missing 1 values. 02/16/26 15:22:55: Missing time: 2026-02-15 18:00:00+00:00 02/16/26 15:22:55: Warning:: Prediction failed due to lack of data 02/16/26 15:22:55: Exception message: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/16/26 15:22:55: Full stack trace: raise Semaphore_Data_Exception(f'DateRangeValidation Failed in Data Gatherer! \n[Series] -> {series}') exceptions.Semaphore_Data_Exception: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/16/26 15:22:56: Model magnolia_12 FAILED - Null result inserted 02/16/26 15:22:56: Failed results: 02/16/26 15:22:56: dataValue dataUnit timeGenerated leadTime 02/16/26 16:22:54: DateRangeValidation: Series [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: [TimeDescription] -> fromDateTime: 2026-02-15 16:00:00+00:00, toDateTime: 2026-02-16 16:00:00+00:00. interval: 1:00:00, stalenessOffset: None is missing 1 values. 02/16/26 16:22:54: Missing time: 2026-02-15 18:00:00+00:00 02/16/26 16:22:54: Warning:: Prediction failed due to lack of data 02/16/26 16:22:54: Exception message: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/16/26 16:22:54: Full stack trace: raise Semaphore_Data_Exception(f'DateRangeValidation Failed in Data Gatherer! \n[Series] -> {series}') exceptions.Semaphore_Data_Exception: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/16/26 16:22:54: Model magnolia_12 FAILED - Null result inserted 02/16/26 16:22:54: Failed results: 02/16/26 16:22:54: dataValue dataUnit timeGenerated leadTime 02/16/26 17:23:06: DateRangeValidation: Series [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: [TimeDescription] -> fromDateTime: 2026-02-15 17:00:00+00:00, toDateTime: 2026-02-16 17:00:00+00:00. interval: 1:00:00, stalenessOffset: None is missing 1 values. 02/16/26 17:23:06: Missing time: 2026-02-15 18:00:00+00:00 02/16/26 17:23:06: Warning:: Prediction failed due to lack of data 02/16/26 17:23:06: Exception message: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/16/26 17:23:06: Full stack trace: raise Semaphore_Data_Exception(f'DateRangeValidation Failed in Data Gatherer! \n[Series] -> {series}') exceptions.Semaphore_Data_Exception: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/16/26 17:23:07: Model magnolia_12 FAILED - Null result inserted 02/16/26 17:23:07: Failed results: 02/16/26 17:23:07: dataValue dataUnit timeGenerated leadTime 02/16/26 18:22:48: DateRangeValidation: Series [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: [TimeDescription] -> fromDateTime: 2026-02-15 18:00:00+00:00, toDateTime: 2026-02-16 18:00:00+00:00. interval: 1:00:00, stalenessOffset: None is 1 values. 02/16/26 18:22:48: Missing time: 2026-02-15 18:00:00+00:00 02/16/26 18:22:48: Warning:: Prediction failed due to lack of data 02/16/26 18:22:48: Exception message: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/16/26 18:22:48: Full stack trace: raise Semaphore_Data_Exception(f'DateRangeValidation Failed in Data Gatherer! \n[Series] -> {series}') exceptions.Semaphore_Data_Exception: Error Code: -1 DateRangeValidation Failed in Data Gatherer! [SeriesDescription] -> source: LIGHTHOUSE, series: dWnDir, location: PortLavaca, datum: None, dataIntegrityDescription: None, verificationOverride: None, timeDescription: 02/16/26 18:22:49: Model magnolia_12 FAILED - Null result inserted 02/16/26 18:22:49: Failed results: 02/16/26 18:22:49: dataValue dataUnit timeGenerated leadTime 02/16/26 19:23:00: Model magnolia_12 completed successfully ✓ 02/16/26 19:23:00: Results inserted: 02/16/26 19:23:00: dataValue dataUnit timeGenerated leadTime 02/16/26 20:23:00: Model magnolia_12 completed successfully ✓ 02/16/26 20:23:00: Results inserted: 02/16/26 20:23:00: dataValue dataUnit timeGenerated leadTime 02/16/26 21:22:56: Model magnolia_12 completed successfully ✓ 02/16/26 21:22:56: Results inserted: 02/16/26 21:22:56: dataValue dataUnit timeGenerated leadTime 02/16/26 22:22:51: Model magnolia_12 completed successfully ✓ 02/16/26 22:22:51: Results inserted: 02/16/26 22:22:51: dataValue dataUnit timeGenerated leadTime 02/16/26 23:22:48: Model magnolia_12 completed successfully ✓ 02/16/26 23:22:48: Results inserted: 02/16/26 23:22:48: dataValue dataUnit timeGenerated leadTime [semaphore.svc@sherlock-dev magnolia_12]$ | | :---- |

On February 16th, all three Magnolia models (magnolia_12, magnolia_24, magnolia_48) experienced 19 consecutive failures between 00:22 and 18:22 CST, caused by two distinct data issues.

The 00:22 and 03:22 CST runs failed on dWl (water level) at Port O'Connor, which was missing its most recent hourly observation at query time — 2026-02-16 00:00 UTC and 2026-02-16 03:00 UTC respectively. Querying LIGHTHOUSE today confirms both timestamps are now present with valid values, indicating these were data latency failures: the observations existed at the sensor level but had not yet been published to LIGHTHOUSE when Semaphore queried. This is consistent with the latency failure observed on February 10th at 02:22 CST.

The remaining 17 runs (01:22–18:22 CST) all failed on dWnDir (wind direction) at Port Lavaca, missing the 2026-02-15 18:00 UTC observation. LIGHTHOUSE data for that timestamp shows wsd and wdr as NA for Port Lavaca, confirming this is a permanent sensor dropout identical in nature to the February 9th 18:00 UTC gap that caused widespread failures on February 10th. The data was never recorded and cannot be recovered.

The Magnolia failures fall into two categories that need different solutions:

For the permanent sensor dropouts on dWnDir): For the Magnolia Models we don’t currently interpolate dWnDir or dWnSpd so we could alter the magnolia models to allow interpolation on this series. For the south Bird Island models, Bird Island interpolation is working correctly — it successfully handles interior gaps throughout the day, but cannot recover the single run where the missing timestamp falls exactly on the window boundary. This is an inherent limitation of limit_area: inside rather than a misconfiguration. So to fix this, we can maybe do something similar to what we did with NDFD before where we try to grab more data than we actually need? Open to Discussion!

For the (assumed) latency failures (Port O'Connor dWl arriving late): We could move the time to further past the hour? The issue here is that we have so many models running already that something like a retry which would rerun the model after 10/15 minutes, seems repetitive when the model is going to run again in the next hour. I’m also hesitant to run further past the hour when it seems like most of the time we don’t have this issue… Open to discussion!

SEMAPHORE SOURCE — MAGNOLIA_TRANSFORM CASCADE

	This is not an independent data source failure The 'SEMAPHORE' source in these failures refers to magnolia_transform models reading output produced by magnolia_12/24/48. When magnolia fails due to LIGHTHOUSE gaps, its output is absent and magnolia_transform fails downstream. This is a dependency cascade, not a separate ingestion problem.

Dependency chain:

LIGHTHOUSE raw data source	→	magnolia_12/24/48 base models	→	magnolia_transform_12/24/48 reads SEMAPHORE (magnolia output)

Model success rates

MODEL	TOTAL RUNS	FAILURES	SUCCESS RATE	PRIMARY FAILURE SOURCE
magnolia_transform_48	260	260	0.0%	SEMAPHORE + LIGHTHOUSE
magnolia_transform_12	260	235	9.6%	SEMAPHORE
magnolia_transform_24	260	226	13.1%	SEMAPHORE
magnolia_12	259	145	44.0%	LIGHTHOUSE
magnolia_24	259	145	44.0%	LIGHTHOUSE
magnolia_48	259	145	44.0%	LIGHTHOUSE

magnolia_transform_48 is at 0% success for the entire reporting period. Even on days when magnolia_12 and magnolia_24 occasionally succeed, magnolia_transform_48 never does. Its lookback window for lighthouse data is large enough that it never has enough un-stale lighthouse data, there’s already a ticket up to fix this that hasn’t been pushed.

NOAATANDC — MISSING DATA ANALYSIS

NOAATANDC accounts for 72 missing data failures spread across 11 days. Unlike LIGHTHOUSE, there is no single catastrophic event — failures are smaller in scope (typically 1–3 models per gap) but occur almost daily.

Possible NOAA update window pattern Failures cluster heavily around 00:00-01:00 UTC and again at 20:00-21:00 UTC. For VirginiaKey models specifically, the missing timestamp is often exactly 01:00 UTC on multiple consecutive days (Feb 8, 9, 10, 11). This is consistent with a scheduled feed where one slot occasionally arrives late — worth checking whether NOAA's TANDC feed has a known update cycle around these times.

Further investigation is required but I decided to look into Lighthouse issues first.

NOAATANDC failures by hour (UTC)

![][image5]

MODEL GROUP	FAILURES	NOTES
96hr_VirginiaKey_wl	9	Longest forecast window; most sensitive to any gap
nj_mlp_surge_*	7-8	All four horizons affected roughly equally
ar_inundation_apr_*	2-6	Seasonal inundation models
pi/rp_mlp/S2S_*	1-6	Scattered; not consistently same gap timestamps
72hr_VirginiaKey_wl	5
48hr_VirginiaKey_wl	2	Feb 10 multi-hour gap (07-12 UTC)

Feb 10 was the worst NOAATANDC day (21 failures). This was a multi-hour gap between approximately 07:00–12:00 UTC, causing rolling failures across several model run windows rather than a single timestamp miss.

NDFD_JSON — STALE DATA EVENT

	Single isolated event — Feb 10, 05:00–06:00 UTC All 62 NDFD_JSON stale failures trace back to one two-hour window on Feb 10. 29 models failed at 05:00 and 06:00 UTC due to stale NDFD data. By 07:00 only 2 failures remained and the issue was fully resolved. This does not appear to be a recurring pattern in this dataset.

TIME (UTC)	MODELS FAILED	FAILURE TYPE
2026-02-10 05:00	29	stale_data
2026-02-10 06:00	29	stale_data
2026-02-10 07:00	2	stale_data
2026-02-09 17:00	2	stale_data

All NDFD_JSON failures affected the Bird-Island Water Temperature forecast models (all horizon lengths) plus a few surge models. The simultaneous multi-model failure is consistent with NDFD publishing a delayed or cached forecast file rather than an individual model ingestion issue.

MODEL HEALTH SUMMARY

All 6 models below 80% are magnolia variants. 35 models ran perfectly for the full 11-day period.

SUMMARY & DIAGNOSTIC NOTES

	Root cause: LIGHTHOUSE reliability is the single biggest lever Fix LIGHTHOUSE gap frequency and you simultaneously fix: all Bird-Island/MRE failures, all magnolia failures, all magnolia_transform cascade failures, and ThermalRefuge failures. That accounts for ~91% of all failures in this dataset.

	NOAATANDC failures are low-volume but consistent Daily occurrence at roughly the same times (00-01 UTC, 20-21 UTC) suggests a recurring late-delivery issue from NOAA's TANDC feed. A small fetch-retry window around known delay times could eliminate most of these.

	NDFD_JSON stale event appears isolated No recurring pattern observed in this window. Monitor for recurrence but no immediate action needed based on this data alone.

Generated from: semaphore_dev_stats_20260218_135429 csv·26,346 records sherlock-dev Feb 8–18 2026