docs planning RUST_STATION_INGESTION_RST02 - AlexandreZanata/Abastevo GitHub Wiki
RST-02 — Pure Rust streaming parser over local files
Status: IMPLEMENTED — parser library only; no network, no database, no
CLI, no migration. Date: 2026-10-09. Scope:
RUST_STATION_INGESTION_PLAN.md RST-02 against the frozen RST-01
fixtures. Language: English document; user communication in Portuguese.
1. Crate (tools/station-prep/, station-prep 0.1.0)
src/lib.rs— public surface:parse_registry,parse_pmqc,AliasTable,Limits,Counts,RunState, row/batch types.src/cnpj.rs— text validation byte-compatible with the Go kernel (ParseCNPJ): formatting stripped, letters uppercased, leading zeroes preserved, both check digits under the alphanumeric program.src/municipality.rs— bounded Latin fold (diacritics stripped, uppercase, whitespace collapsed) plus versioned alias table; unknown or ambiguous rows quarantine, never guessed.src/registry.rs—;-delimited streaming reader, frozen 13-column name matching, per-field byte cap, membership-as-evidence mapping, SHA256 canonical checksums, within-run dedup, succession links and stable(source_key, checksum)output order.src/pmqc.rs— sample-identity dedup, empty coords stay accepted without a point, zero/non-finite/out-of-bounds quarantine, mirrored pairs staysuspect_swap(never auto-swapped), originalEPSG:4674withaccuracy_m: nullandreview_state: pending.src/types.rs— bounds, reconciled counts, run states, frozen typed quarantine rows withfile:logical-recordlocators.tests/parser.rs— 9 integration cases over the RST-01 fixtures; unit vectors live beside each module.
2. Contract deltas from RST-01 (evidenced, minimal)
- Framing rule: a complete snapshot ends with a record terminator.
The
csvreader tolerates EOF inside quotes, so a cut mid-quote would otherwise decode as a shorter valid file; boundary-exact cuts stay the Go loader's manifest-count check (RST-05). - One reason added:
invalid_fieldfor malformed non-identity fields (empty business name, malformed UF, empty sample reference). All RST-01 reasons are unchanged. - Normalization is a bounded Latin fold, not full NFKD; it covers the fixture/reference sample and every IBGE Portuguese name class. Full NFKD awaits a measured need.
staleness_noteis always empty from the parser; the pure parser has no clock and freshness review belongs to the Directory owner.- Succession: same CNPJ with several snapshots keeps every row; the
newest publication carries
supersedes: nulland older rows link to its checksum. Older evidence never overwrites fresher facts.
3. Dependencies, MSRV, licences, audit
Cargo.lock is tracked. Direct pins: csv 1.4.0, serde 1.0.229
(derive), serde_json 1.0.151, sha2 0.10.9; 23 locked transitive
crates, all permissive (MIT / Apache-2.0 / Unlicense/MIT /
BSL-1.0 for ryu). rust-version = "1.99" (tested toolchain
rustc/cargo 1.99.0); no older MSRV is claimed. cargo audit 0.22.2
over 1296 advisories: no vulnerabilities, exit 0. No network, process
or unsafe code in src/ (verified by search). No Tokio/reqwest/
parallel pools/PostgreSQL client/dataframe.
4. Evidence (RED → GREEN, revision 157d476 + working tree)
- RED:
cargo testfailed with no library source; then 3 implementation failures (borrowck in succession grouping, wrong test field paths, flawed reorder/truncation test framing) fixed with evidence. - GREEN:
cargo test --locked— 4 unit + 9 integration passed. Registry sample evaluates 12 = 7 accepted (one superseded history) + 1 duplicate + 4 quarantined; PMQC 11 = 5 + 1 + 5. Reordered inputs yield byte-identical accepted JSONL; BOM strips; header variant quarantines the run; truncation fails with no partial output; emitted JSONL key sets equal the frozen samples. cargo fmt --checkclean;cargo clippy --all-targets -- -D warningsclean. Existing Go registry contract tests re-passed in RST-01 and are untouched here.- Measured peak RSS of the integration binary on synthetic fixtures:
3.5 MiB (
/usr/bin/time -v, max RSS 3548 kB). No national-scale or speedup claim; laboratory targets stay RST-07 business.
5. Next task: RST-03 (smallest)
Determinism and deltas on top of this parser: canonical hashing policy shared with Go, external sorted spill/merge past the in-memory cap, unchanged-replay no-op proofs, supersession across batches and interrupted-output handling. Acceptance: reordered national-shaped synthetic input yields identical output bytes, corrected bytes yield dated corrections, older evidence never wins, and disk-spill behavior is measured, not asserted.