eeu.tmc — TMC Traffic-Data File

Status: 🟡 Partially cracked · 25.8 MB · NOT_COMPRESSED · File header, the 44-table directory, AND the complete per-location byte-offset index (100% of the file's own addressing structure, validated at full scale, zero exceptions) are cracked; 2 bytes of the leaf chain records' own content (bytes 2 and 4 — each 2 independent bit-packed sub-fields, candidates start/side and a matched exception-like flag pair) are now bit-pinned too, but most of the rest of that content remains undecoded.

The last "genuinely unexplored" NOT_COMPRESSED file on the disc, cracked using ground truth found via config/create_cd: telemat/tmc2/ TMCCONFIG.ini confirmed this disc's real ALERT-C/TMC conventions, and dbal/'s own embedded source-file list independently confirms the real handler is named db_tmc.cpp (plus a db_tmc_deprecated.cpp twin — CORRECTED, a later session's 5-version diff of every dbal/V0*/DBAL.OUT: both files coexist in ALL 5 DBAL versions, refuting the earlier guess that the on-disc TMC format changed between DBAL releases; _deprecated more likely means a legacy/alternate encoding kept for backward compatibility, see the dbal/ page for the full 5-version diff). Independently confirmed a 2nd way, a still-later session, from the FIRMWARE side (a separate factory firmware disc the user provided): the firmware's own embedded debug strings name the real versioned accessor pair directly, db_tmc_all_headers_V003 (current) and db_tmc_all_headers_ deprecated_V005 (deprecated), plus db_tmc_seg_V003 — i.e. _V003/ _V005 are real ON-DISC TMC FORMAT VERSION numbers the firmware's own dbaLib code branches on per-disc, not DBAL BUILD versions — the same conclusion the 5-version diff already reached, now confirmed directly from the accessor names themselves. See the main README §2.6 (firmware findings, out of scope for this map-disc-focused wiki) for the full firmware-side picture.

eeu.mod's real schema for this table

eeu.mod names this table "tmc file" (74 raw strings). After the usual boilerplate, the real structure is:

tmc_file_header:
    no_of_location_tables
    max_segment_chain_len
    max_location_size
    location_table_headers[] -> location_table_header:
        name
        start_location
        no_of_locations
        offset_of_location_offset_table_p
        offset_of_location_offset_table_n
location_offset_tables:
    location_offset_table_p[] -> {chain_offset, start, location_p_end}
    location_offset_table_n[] -> {chain_offset, start, location_n_end}
location_chains:
    location_chains_p / location_chains_n -> segment_chains_for_one_location:
        chain_count, total_no_of_chains, no_of_internal_chains
        segment_chains[] -> segment_chain:
            start, vseg_id, side, no_of_segments
            exception: {start_exception_arm_no, start_exception_valid,
                        end_exception_arm_no, end_exception_valid}
            exploration_points[] -> exploration_point: {arm0, arm1, offset}

Cracked, validated at full scale: the file header + 44-table directory

uint16 LE  no_of_location_tables  -- 44, exactly matching the number
                                      of directory records that follow
uint16 LE  ???                    -- 1716; position matches
                                      max_segment_chain_len, value not
                                      independently confirmed
uint16 LE  ???                    -- 240; position matches
                                      max_location_size, not confirmed

44 x 20-byte location_table_header records:
    bytes[4]   code         -- 3-char alphanumeric + NUL (e.g. "117",
                                "225", "A01", "F49") -- confirmed
                                exactly 3 characters + NUL on all
                                44/44 records
    uint32 LE  start_location?    -- position matches, not confirmed
    uint32 LE  no_of_locations?   -- position matches, not confirmed
    uint32 LE  offset_p           -- cracked (identity), see below
    uint32 LE  offset_n           -- cracked (identity), see below

Validated: the 44-record count exactly matches the header's own no_of_location_tables field. offset_p/offset_n are monotonically increasing across all 44 records (980, 105460, 209940, 452364, 838444, ..., up to 8,482,780) with offset_n > offset_p on every record, and each record's own offset_p at or past the previous record's offset_n — exactly the shape real, sequentially-allocated byte offsets into a shared payload would have, matching eeu.mod's own field names by position exactly. Spot-checked: the bytes AT both offset_p[0]=980 and offset_n[0]=105460 are themselves further small, regular, non-random binary structure, not garbage/misalignment.

A real, unexplained sub-pattern: 4 consecutive tables (codes 714- 717) share the exact same start_location? value (10001) — plausibly sub-divisions of one larger region.

The 44 codes are NOT the same identifier convention telemat/tmc2/ TMCCONFIG.ini's own 14 Western-European countries use (hex Country Code + decimal LTN) — this disc's own 44 tables are a real, distinct catalog, plausibly covering the "eeu" (East Europe) dataset's own wider country/region footprint. Not matched against external ground truth.

A later session: the COMPLETE per-location byte-offset index CRACKED — validated at 100% scale, zero exceptions

Each table's own offset_p doesn't point at a single struct — it's the start of a flat array, eeu.mod's own location_offset_table_p. Its element count is exactly no_of_locations + 1 (the classic prefix-sum/ CSR shape), confirmed EXACTLY on all 44/44 tables, zero exceptions:

(offset_n - offset_p) / (no_of_locations + 1) == 4   -- for all 44/44 tables

i.e. location_offset_table_p is no_of_locations + 1 little-endian uint32 absolute file byte offsets, monotonically non-decreasing on every table (confirmed, zero exceptions) — location i's own data spans [table_p[i], table_p[i+1]).

location_offset_table_n (the mirror "negative direction" array) immediately follows location_offset_table_p, starting at offset_n and using the IDENTICAL (no_of_locations + 1) × 4-byte size — confirmed EXACTLY on all 43/43 consecutive table pairs:

next_table.offset_p - this_table.offset_n == (no_of_locations + 1) * 4

The whole file is now fully self-consistent end to end: the LAST table's (F49) own location_offset_table_n — unbounded by a "next" directory record — computes to end at exactly byte 8,584,500, and separately the FIRST table's (117) own location_offset_table_p[0] value is ALSO exactly 8,584,500 — 2 independent computations agreeing to the byte. The LAST value of the LAST table's own location_offset_table_n equals 25,814,096 — the file's own exact total size (EOF), also to the byte. Every offset for every one of the 44 tables falls within the file's real bounds, and both arrays are monotonic on every table — no exceptions found anywhere.

Practical consequence: byte 94 (end of the SIEMENS header) to EOF is now fully mapped:

[6-byte tmc_file_header]
[880-byte, 44-record directory]
[44 x (location_offset_table_p + location_offset_table_n) pairs]   -- 886 to 8,584,500
[shared "location_chains" bulk region]                              -- 8,584,500 to EOF (17,229,596 bytes, 66.75% of the file)

Given any (table_code, location_index), that location's exact byte range in BOTH ALERT-C directions is now directly computable, with zero guessing.

A later session: chain_count cracked, and the 7-byte-minimum segment_chain shape confirmed exactly

Individual locations are small (single digits to a few dozen bytes) and eeu.mod's own nested schema for them (segment_chains_for_one_location: chain_count, total_no_of_chains, no_of_internal_chains, segment_chains[] -> segment_chain: {start, vseg_id, side, no_of_segments, exception{...}, exploration_points[]}) is considerably deeper than the flat offset index above — variable- length nested arrays, not another fixed-shape table.

CRACKED: byte 0's low nibble = chain_count, confirmed by an EXACT integer formula, not just a trend. Grouping every non-empty location record by byte 0's low nibble and taking each group's own MINIMUM record size:

low nibble	n	min size	1 + n×7
1	724,059	8	8 ✓
2	493,817	15	15 ✓
3	691	22	22 ✓
4	83	29	29 ✓
6	8	43	43 ✓

Exact match on every group tested (the one gap, low_nibble=5 with only 29 samples, missed the true minimum by 1 byte — a sampling gap, not a contradiction). This confirms a fixed 1-byte record header and a 7-byte MINIMUM footprint per segment_chain (reached at 0 exploration_ points) — mean size also grows roughly linearly with chain_count (10.4, 19.5, 28.8, 39.4, 50.8, 57.5 bytes for nibble 1-6).

Byte-level characterization (statistical, NOT bit-pinned) of the minimal 7-byte segment_chain, from the FULL, true-scale chain_count=1, exactly-8-byte-total subset (310,888 records, all 44 tables, both p/n directions).

Self-correction (same session): an earlier pass of this exact analysis used an accidentally truncated ~26,220-record sample (only the first few tables scanned, due to an early-break condition) and wrongly reported byte 4 as "mostly constant, bimodal 18/146" and byte 6 as "always 0, refuted as an exploration_points count". Both are corrected below, re-run without the early break. A reminder: always verify a sample is representative before trusting a "mostly constant" finding on this disc.

• byte 0: only 2 distinct values (1, 17) — low nibble=chain_count, bit 4 a 2nd flag, set on 72.1% of records
• bytes 1, 3: high entropy (256 distinct values each) — candidates for (part of) vseg_id, real meaning still open. Tested and REFUTED (a later session), 2 hypotheses for the former "bytes[1:4] as one field" view: as a plain within-table location index (r ≈ 0.012, essentially zero correlation across 2,000 real records) and as a cross-reference (big-endian) to another location's own offset-table boundary elsewhere in the file (only 2.86% of 74,848 real records land on one of 1,218,695 real boundary offsets — below the ~7% rate pure chance predicts given that set's own density, i.e. coincidental collisions). Neither holds. A lead raised, then REFUTED (both the same still-later session): byte 1's high nibble is NOT uniform like byte 2's arm candidate — it decays roughly geometrically from ~26,000 (nibble=0) down to ~14,000 (nibble=15), while its own low nibble IS roughly uniform (~19,000-20,400 each) — the classic shape of a HIGH byte of a larger little-endian value with a real-world -skewed (smaller-is-commoner) distribution, suggestive of byte 1 being (part of) vseg_id itself, with byte 3 as a candidate low byte. Tested directly against real ground truth and refuted: combined as a 16-bit value (both byte orders tried) and checked against eeu.si's own real, confirmed seginfoID range (0-24,352, 24,353 records, research/si_reader.py) — if vseg_id really were a direct cross-reference into eeu.si, most combined values should land in that range. Instead only 35.9%-44.7% did, matching the ~37.2% rate pure chance alone predicts for a value spanning the observed full 0-65,534 range — not a real signal. vseg_id is not a direct eeu.si index via this byte pairing; byte 1's own skewed shape remains real but unexplained.
• byte 2 — CRACKED (a later session): 2 independent bit-packed sub-fields, not part of one 3-byte blob. eeu.mod's schema lists start and vseg_id as separate fields — the earlier "bytes[1:4] as one blob" framing never actually tried splitting them. Byte 2's own 48 distinct values are EXACTLY (high_nibble << 4) | low_nibble: high nibble spans all 16 values 0-15 (count ~12,250-13,100 each — uniform), low nibble only ever 0, 1, or 2 (~65%/33%/2% — skewed, but identically skewed in both the p and n direction, ruling out redundancy with the file's own already-known p/n split). 16 × 3 = 48 — an exact combinatorial match, zero violations across all 310,888 real minimal records, independently consistent in every one of the 44 tables. The two sub-fields are also statistically independent of each other — the 3-value field's distribution is ~65%/33%/2% at every one of the 16 high-nibble values (sampling noise only) — the classic signature of two separately-assigned fields sharing a byte, not a coincidental split of one combined value. Best real-world candidates by schema position and range: high nibble = start (a junction "arm" index, 0-15 — ALERT-C locations commonly have few enough arms to fit a nibble), low 2 bits = side (a small, skewed real-world category, plausibly "no divided carriageway / right / left" — most roads being undivided matches the ~65% dominant value). Neither candidate name is independently confirmed yet (no external ground truth checked against), but the bit-level split itself is exact, not a trend.
• byte 4 — HIGH NIBBLE CRACKED (same later session), same shape as byte 2. 96 distinct values total (= 6 × 16 — matches the crack exactly). The high nibble takes only 6 discrete values, {0,1,2,8,9,10}, zero violations across all 310,888 real minimal records: exactly (flag << 3) | val3 with flag a real ~50/50 binary bit (matches the page's earlier "bit 7" observation) and val3 ANOTHER 3-valued field (0, 1, 2) whose own marginal distribution (~65.6%/32.7%/1.7%) is nearly identical to byte 2's "side" candidate — yet cross-tabulation shows the two are statistically independent (side's own proportions stay ~65/33/2% at every val3 value, and vice versa). Two different 3-valued fields sharing almost the same real-world distribution, but provably not the same information twice, is consistent with a matched START/END pair — eeu.mod's exception: {start_exception_arm_no, start_exception_valid, end_exception_arm_no, end_exception_valid} is the obvious candidate structure, though which specific sub-field maps to which bit isn't confirmed. The byte's low nibble (16 genuinely distinct, uneven values) is not yet cracked.
• byte 5: power-law distributed, 78.6% == 1 (robust at full scale) — matches real-world no_of_segments (most chains cover exactly 1 road segment)
• byte 6: 99.955% == 0, but a REAL, rare (325/724,059 = 0.045%) nonzero signal exists (values 1-4) that correlates STRONGLY and MONOTONICALLY with how much extra content the record has past the 8-byte minimum: mean extra bytes 2.41 (byte6=0), 41.88 (=1), 75.00 (=2 and =3, tied), 225.00 (=4, single sample) — consistent with a real internal-chain- or exploration-point-related count that's simply almost always 0, NOT refuted as the small sample originally suggested
• byte 7: 41 distinct values — splits into a high nibble overwhelmingly concentrated at 8/9/10 (~97% of records) and a low nibble concentrated at 8/9 (~82%) — the classic shape of a small value stored with an "excess-8" bias (centered near 0 after subtracting 8). Checked (a still later session) against the 2 already-cracked fields: no correlation at all with the arm candidate (byte 2's high nibble) — mean high-nibble ~8.70-8.72 across every one of the 16 arm values, flat; only a weak, not clearly significant upward trend against no_of_segments (byte 5: mean ~8.70 at no_of_segments=1 rising to ~8.7-8.85 by no_of_segments=7-10, but those higher groups are much smaller samples). Inconclusive, not a crack — real meaning still open, including whether it's related to exploration_point-style small signed deltas even though minimal records (by construction) have 0 real exploration points, so if real, it'd be some other small signed field.

Bytes 2 and 4 now reach the same bit-pinned standard as this disc's other fully-cracked fields (e.g. eeu.si's rank/class/ divided) — exact, validated at full scale, not a trend. Bytes 1, 3, and 7 remain only statistically characterized, not individually confirmed.

Reader code

research/tmc_reader.py — read_tmc_directory(), read_location_offset_arrays(), location_byte_range(), read_location_chain_bytes() (hands back the exact raw bytes for any location), chain_count() (the cracked chain-count field), decode_segment_chain_byte2() (the cracked (arm_candidate, side_candidate) sub-fields), decode_segment_chain_byte4() (the cracked (flag, val3) sub-fields).