docs release evidence p25 t02 csv staging - AlexandreZanata/Abastevo GitHub Wiki
P25-T02 — Bounded CSV snapshot staging
Status: LOCAL_DONE on codex/phase-25-national-registry. Task: P25-T02.
Binds B-BR-D02/D03/D05/D10 and BUC-D01/D06. Staging only: bounded
parser + run ledger/checksum/checkpoints; publication stays P25-T04.
Append-only migration; no station/price/trust writes.
Behavior (TDD, critical-first)
- Migration
000031_registry_sources.sql(append-only):registry_source_runs(source/snapshot/checksum/parser/state + accepted/duplicates/rejected/error_code) with(source, snapshot_identity)uniqueness, andregistry_assertions(stablesource_key= CNPJ text, row checksum, validated address/ status/eligibility fields, nullablestation_idfor T04 resolution) with(source, source_key, checksum)replay idempotence. Publishers read complete runs only. (authorizationrenamedauth_state: reserved SQL keyword, caught bysqlc vet.) - sqlc: directory stanza extended with the new migration;
db/queries/directory/registry.sql(create/get/finish run, stage assertion withON CONFLICT DO NOTHING, counts);sqlc vet+generateclean, no generated diff. registry/stage.goStageCSV: bounded read (100 MB cap → failedoversizerun), sha256 snapshot checksum, create-or-join run (replay returns the existing report; concurrent stagers converge via the assertion key), header-name detection (missing/unknown → quarantined run, zero assertions), per-row validation (kernel CNPJ text, required identity/address, status→auth/eligibility mapping), row sha256, short batches, full accounting (accepted+duplicates+rejected = rows seen). Truncated input, row-cap breach and stage errors fail the run explicitly; empty input quarantines; the last-good publication is never touched (staging writes nothing outside its own tables).- Fixed along the way from real failures (not weakened): reserved
column rename,
CreateRegistryRunno-row-on-conflict handling, duplicate counting viaRETURNINGabsence, made-up CNPJs replaced by check-valid synthetic values.
Validation
- Unit:
registrypackage 11/11 PASS (-race): header require/unknown/missing, CNPJ preserve/reject, status freeze, eligibility matrix, manifest markers + no-PII, row accounting, replay idempotence, quarantine/oversize/truncated/empty/row-cap/ stage-error paths (fake store). - Integration (real PostGIS,
-race -tags=integration): replay + 4-way concurrency converge on 3 assertions; manifest sample quarantines end to end with 0 assertions; truncated preserves the single last-good complete run; empty quarantines with 0 complete runs. Fresh disposable DBs prove the append-only 000031 upgrade on previous schemas (empty/upgrade/recovery). - Regression: full
directory/...unit + integration PASS (8 pkgs each);go vetclean;vacuum lint+apicontractPASS (contracts untouched). git diff --checkPASS;scan-secrets.shPASS (no PII/secrets; all CNPJs synthetic[P25-TEST]).
Limits and next
- No live ANP fetch was performed (sandbox egress + provisional columns); national-scale calibration stays P29-owned.
- Next: P25-T03 paginated/targeted ANP API discovery on this branch.