integrity oracle - XibalbaTechSol/integrity-core GitHub Wiki

title: integrity-oracle created: 2026-07-07 updated: 2026-09-18 type: entity tags: [infrastructure, metrics, layer-2, tokenomics] confidence: high source_files:

• integrity-oracle/scoring-core/src/lib.rs
• integrity-oracle/backend/src/handlers.rs
• integrity-oracle/backend/src/derive.rs
• integrity-oracle/backend/src/otlp.rs
• integrity-oracle/backend/src/chain.rs
• integrity-oracle/backend/src/config.rs
• integrity-oracle/backend/src/db.rs
• integrity-oracle/backend/src/phi.rs
• integrity-oracle/backend/src/vc.rs
• integrity-oracle/backend/src/verification.rs
• integrity-oracle/backend/src/attestation.rs
• integrity-oracle/backend/src/kyc.rs
• integrity-oracle/backend/src/crypto/mod.rs
• integrity-oracle/backend/src/openapi.rs
• integrity-oracle/backend/migrations/0001_init.sql
• integrity-oracle/backend/migrations/0002_markets_and_judge.sql
• integrity-oracle/backend/migrations/0003_agent_did_document.sql
• integrity-oracle/backend/migrations/0021_otel_retention.sql

The off-chain brain (Rust, Axum, Postgres, Redis, alloy): it ingests agent telemetry, computes the Agent Integrity Score, and independently verifies agents' on-chain state — including, as of this pass, the market/application layer (§6.9) — so nothing downstream has to trust an agent's own word. Per §6.10 of the interface contract, this is the only backend that ever reads on-chain state.

• Workspace
• HTTP API
  • GET /v1/agent/{id}/telemetry, GET /v1/agent/{id}/traces
  • GET /v1/markets, GET /v1/markets/{id} (§6.9)
  • GET /v1/leaderboard
  • Contract-ownership + Integrity Health reads: GET /v1/agent/{id}/contracts, /baas
  • Verifiable Credentials: GET /v1/agent/{id}/vc
  • Network benchmarks + protocol stats: GET /v1/benchmarks, /v1/stats
  • XNS handle directory (Oracle-local, primary): GET /v1/xns/available, GET /v1/xns/handle/{handle}, POST /v1/xns/claim
  • On-chain XNS alias resolution (secondary): GET /v1/xns/resolve?handle=, GET /v1/agent/{id}/handle
  • Governance: GET /v1/governance/proposals
  • GET /v1/agent/{id}/wallet
  • Server-side telemetry-signal re-derivation (derive.rs)
  • Signed telemetry schema v3 (handlers.rs + spanschema.rs)
  • The OTLP/gRPC path (otlp.rs) — separate from telemetryevents, unauthenticated
  • PHI backstop on POST /v1/telemetry/ingest
  • Judge evaluations (storage only — no judge implementation)
• On-chain client (chain.rs)
• Anchoring
• Canonical JSON signing — RFC 8785 JCS
• Finality-bearing agent directory
• State

• scoring-core — dependency-free; the one place the AIS formula is computed. Everyone else reads the HTTP API.
• backend — Axum server, sqlx persistence, Redis rate limiting, bb verify ZK verification, Merkle building, and the alloy on-chain read client.

POST /v1/agent/register
GET  /v1/agent/{id}
GET  /v1/agents
GET  /v1/agent/{id}/ais
GET  /v1/agent/{id}/ais/history
GET  /v1/agent/{id}/compliance
GET  /v1/agent/{id}/wallet
GET  /v1/agent/{id}/telemetry
GET  /v1/agent/{id}/telemetry/volume
GET  /v1/agent/{id}/otel/volume
GET  /v1/agent/{id}/traces
GET  /v1/agent/{id}/contracts
GET  /v1/agent/{id}/baas
GET  /v1/agent/{id}/vc
GET  /v1/agent/{id}/handle
POST /v1/agent/{id}/verify/dns/challenge
POST /v1/agent/{id}/verify/dns
POST /v1/agent/{id}/verify/github/challenge
POST /v1/agent/{id}/verify/github
POST /v1/agent/{id}/verify/tee/challenge
POST /v1/agent/{id}/verify/tee
POST /v1/agent/{id}/verify/kyc/challenge
POST /v1/agent/{id}/verify/kyc
GET  /v1/agent/{id}/verify
POST /v1/agent/{id}/verify/{verification_id}/revoke/challenge
POST /v1/agent/{id}/verify/{verification_id}/revoke
GET  /v1/traces/{trace_id}
POST /v1/telemetry/ingest
GET  /v1/markets
GET  /v1/markets/{id}
GET  /v1/leaderboard
GET  /v1/benchmarks
GET  /v1/stats
GET  /v1/xns/resolve?handle=<h>
GET  /v1/governance/proposals
GET  /v1/stream
GET  /v1/agent/{id}/stream
GET  /healthz

Full request-pipeline order for POST /v1/telemetry/ingest (PHI scan → agent lookup → rate limit → signature → server-side re-derivation → ZK → compliance → nonce replay → storage): Telemetry Ingestion Pipeline.

Real history queries returning lists of past telemetry events and judge traces respectively, queried directly from Postgres (telemetry_events and judge_evaluations tables) and returned as structured DTO arrays. Used by the Dashboard UI to render live feeds and historical execution logs.

The honesty crux: POST /v1/agent/register independently calls XibalbaAgentRegistry.resolveDID and rejects (400) if the client's claimed 7 primitives don't match on-chain state. This is what makes "the chain is the source of truth" real, not decorative. Request body (RegisterAgentRequest, handlers.rs): did (required — not agent_id), did_document, primitives (exactly the 7 PrimitiveSetDto fields), ed25519_pubkey_hex/eth_address_hex (Option<String> each, but the handler 400s if both are absent), verification_tier (i32, defaults to 0). Now documented in docs/INTERFACE_CONTRACT.md §6.3 — it was silent on this schema until 2026-07-09, which is how integrity-sdk's client drifted from it undetected (see integrity-sdk's "Fixed 2026-07-09" note).

On 2026-07-16, we enhanced chain.rs (resolve_primitives_by_did) to log the actual RPC contract call error details when resolveDID fails instead of silently mapping them to UnknownDid. This resolves a major diagnostic visibility gap during stack startup and network resolution.

As of 2026-08-31, every cached primitive read is explicitly scoped to the Oracle's configured chain. The database module intentionally exposes no unscoped primitive-read helper: direct agent reads, fleet enrichment, AIS and compliance calculation, leaderboard refresh, and reverse SovereignAgent-to-DID lookup all require chain_id. Cache rows from another network and legacy rows with chain_id IS NULL are treated as misses rather than evidence about the active chain; the underlying agent identity remains listable. Leaderboard snapshots and freshness markers are keyed by chain and each refresh atomically replaces that chain's derived rows, preventing stale entries from surviving a partial current-chain resolution. This is a cache-isolation property, not proof that a matching-chain cached row is fresh or that the chain itself is available.

Real reads of every IntegrityMarket clone via MarketFactory. Enumeration is concurrent (futures::future::join_all), not a serial loop, and cached in Postgres (markets_cache + a single-row markets_index_sync marker) behind a 30s staleness window (handlers::MARKETS_CACHE_STALENESS_SECS) — a documented tradeoff between RPC load and freshness, not silent staleness. On a cache miss/stale hit, list_markets re-enumerates MarketFactory.allMarketsCount() (not just refreshes existing rows) so a newly-created market is actually discovered.

GET /v1/markets/{id} ({id} = the market's contract address) returns question, outcomeCount, resolved, winningOutcome, resolveDeadline, totalStaked, and outcomeStaked per outcome (the real pari-mutuel pool — cheap public-getter reads). An optional ?agent=0x... query param adds a real, single getPosition read as your_position. Documented gap: enumerating every holder's position across a market requires indexing PositionEntered events, which this pass does not build — positions_note in the response says so explicitly rather than silently omitting it.

Uint256 amounts (min_ais_to_enter, total_staked, outcome_staked[i]) are always decimal strings in both the DB (TEXT, not a numeric type sqlx isn't configured for) and the JSON DTOs — a uint256 can exceed both i64 and f64's safe integer range.

Contract-vs-brief discrepancy found while building this: MarketFactory has no allMarkets() returns (address[]) getter — address[] public allMarkets only auto-generates an indexed allMarkets(uint256) returns (address) getter. Enumeration is allMarketsCount() + concurrent allMarkets(i) reads (ChainClient::all_market_addresses). By-creator listing uses the real getMarketsByCreator(address) returns (address[]) function, not the marketsByCreator mapping's auto-getter (which also needs an index).

Ranks agents by real ReputationRegistry.effectiveScore (decimal string, same ChainClient::effective_score method chain.rs already had). No fabricated P&L: realized_pnl is always null — computing it for real would require indexing IntegrityMarket PositionEntered/MarketResolved/PayoutClaimed events across every market, out of scope for this pass. An honestly-incomplete ranking, not a silent mock. Only agents with a resolvable on-chain PrimitiveSet (cached, or live-resolved and cached on the fly) appear.

/contracts returns the IntegrityMarket clones an agent deployed and owns (MarketFactory.getMarketsByCreator on its SovereignAgent, then live per-market reads). /baas enumerates the SmartBAA escrows where the agent is the business associate, from SmartBAAFactory.BAACreated logs (no reverse index exists on-chain, so the event log is the only enumeration path) + each escrow's live status. Both are real on-chain reads; /baas returns a clean MissingSingleton (400) where the Integrity Health SmartBAAFactory isn't deployed.

Issues a signed W3C Verifiable Credential (AgentIntegrityCredential) for the agent's current AIS + verification tier — a real Ed25519 Ed25519Signature2020 proof over the canonicalized credential, signed by the oracle's issuer key (crate::vc, VC_ISSUER_SEED env, did:key issuer). Not a mock: the returned JSON-LD credential is independently verifiable against the issuer's public key. Consumed by the dashboard's DIDExplorer.

/benchmarks aggregates network-wide telemetry per model (db::benchmark_by_model, GROUP BY payload->>'model' HAVING COUNT >= 3) into a per-model/provider stability + grounding benchmark (a simulated AIS from the same entropy/grounding math scoring-core uses). /stats is the minimal protocol-wide singleton supplement the dashboard can't cheaply derive from its per-agent loop: marketplace volume (sum of cached market total_staked) + A2ACapitalPool escrow/release totals; tvl is composed client-side for a single source of truth.

As of 2026-09-26 (PRODUCTION_GAPS.md #71) this, not the on-chain resolver below, is the primary XNS system: a chat-app-style unique handle per agent, chosen by the user, stored in the oracle's own xns_handles Postgres table (agent_id PK, unique index on lower(handle)), independent of chain state entirely. GET /v1/xns/available?handle=<h> checks availability and returns suggested alternatives when taken; GET /v1/xns/handle/{handle} resolves a handle → agent_id; POST /v1/xns/claim claims a handle for an agent (409 if taken). POST /v1/agent/register now takes a mandatory handle and claims it transactionally with primitive registration (db::register_agent_with_handle) — every agent registered from here on has a handle unconditionally; list_agents sources it via db::get_xns_handles_for_agents. Agents without a handle (registered before this change) are identified by DID. Consumed by XNSSearchService (dashboard) for search, availability, and claim.

Live reads of the XibalbaNameService singleton — now an optional, explicitly secondary alias listing, not the primary handle system (see above). /xns/resolve maps a human handle → SovereignAgent (and, best-effort via db::did_by_sovereign_agent, the reverse DID); resolve_handle gates on handleExists first since the contract's resolve() reverts on an unclaimed handle, so an unregistered handle returns a null address, not an error. /agent/{id}/handle is the reverse: an agent's primaryHandle. Both return MissingSingleton (HTTP 400) until XNS is deployed on the target network — an honest "not deployed", never a fabricated handle. Registering an on-chain alias for an Oracle-claimed handle is not yet built (XNSRegisterForm.tsx, which wrote directly to this contract, was removed 2026-09-26 — see PRODUCTION_GAPS.md #71 — because it bypassed the Oracle-local uniqueness table and could disagree with it about who holds a handle).

Live enumeration of IntegrityGovernance proposals by index (ChainClient::read_proposals: proposalCount() then getProposal(i) + state(i) for i in 1..=count, 1-based, newest-first — an index loop, deliberately not log-scanning). Each DTO carries proposer, target, FOR/AGAINST tallies (decimal-string wei of ITK), the timelock ETA, and the derived state (Active/Defeated/Succeeded/Queued/Executed/Expired/Canceled). Returns MissingSingleton (HTTP 400) until IntegrityGovernance is deployed; the dashboard's GovernancePanel/GuardianPilot degrade to their honest "Not Yet Live" state on that (never a live-but-empty proposal list). See Governance.

Real IntegrityToken.balanceOf(sovereignAgent) read (decimal string), plus open positions: a real getPosition read against every cached market for that agent's SovereignAgent address, filtered to amount > 0 && !claimed. Documented gap: transaction_history is always null — transfer/stake/payout history requires indexing on-chain events (Transfer, PositionEntered, PayoutClaimed, ...), not built this pass.

POST /v1/telemetry/ingest does not trust a client's self-reported derived_signals. It independently recomputes entropy/grounding from the same signed request's raw otel_spans content and accepts sacrifice/compliance only from independently admissible evidence. Token-derived sacrifice and client compliance remain audit-only proxies and fail closed in authoritative AIS when their evidence is absent. Only the oracle's own admissible values feed telemetry_events/AIS — the client's claim is stored purely as an audit-trail comparison. Two polarity/calibration bugs were fixed at this exact call site in the same pass: performance_variance was receiving the SDK's stability-score polarity (1.0=best) into a column scoring-core treats as a true variance (0.0=best) — backwards for every agent until fixed; and gpu_hours_verified now receives an hours-equivalent proxy rather than a pre-normalized [0,1] index, removing a double-log-compression that capped max-sacrifice agents around ~100/1000 instead of ~1000. Full pipeline-order writeup (this is step 5 of an 11-step ordered handler sequence — PHI scan, agent lookup, rate limit, and signature verification all run first): Telemetry Ingestion Pipeline. Formula and trust-model detail: AIS.

The signed POST /v1/telemetry/ingest envelope currently accepts schema versions 1 through 3. Version 3 is the first version whose otel_spans array is structurally validated before the PHI scan and persistence path: the allowlisted shape, metadata/property counts, text and batch bounds, numeric ranges, and covered-entity address format are checked. Unknown or oversized shapes fail closed with a client-visible bad request. The validator does not apply to the separate unauthenticated OTLP receiver and does not make OTLP data eligible for AIS.

The SDK and oracle version constants must move together. Historical envelopes without a schema_version remain valid, while future versions are rejected until the oracle is upgraded. See docs/INTERFACE_CONTRACT.md §4.2a for the wire-level contract.

Lights up the SDK's already-real OTLPSpanExporter/OTLPMetricExporter (telemetry/core.py::init_telemetry, gRPC localhost:4317), which previously exported into a void — nothing listened on that port. Real spans arrive with no Ed25519/secp256k1 signature envelope (unlike POST /v1/telemetry/ingest), so this deliberately does NOT touch telemetry_events/AIS — feeding unauthenticated spans into scoring would let anyone move an agent's score. Trace export is fully implemented (PHI-scanned via the same crate::phi backstop, persisted to a separate otel_spans table, broadcast over SSE); metrics export is accepted (so the SDK's metric exporter gets a real gRPC response) but not yet parsed or persisted.

src/phi.rs mirrors integrity-sdk/integrity_sdk/security/redactor.py's regex categories (PRIVATE_KEY, API_KEY, SSN, CREDIT_CARD, EMAIL, PHONE, MRN) as a server-side defense-in-depth backstop: it recursively scans every JSON string leaf in otel_spans (and an optional judge_evaluation), and rejects (400, AppError::PhiDetected) if any raw, unredacted pattern is found — belt-and-suspenders for a buggy/bypassed client, never a replacement for the SDK's own client-side redaction. Runs before any DB/RPC work in ingest_telemetry, so it fails fast. Only string values are scanned (numeric/structural fields are skipped) — the SDK's redactor never touches those either, so scanning them here would just produce false positives on values nobody was ever going to redact.

TelemetryIngestRequest.judge_evaluation (optional) persists into the new judge_evaluations table (run_id, judge_model, verdict, score, rationale_summary, linked to the telemetry event). Deliberately NOT part of the signed envelope — ingest_telemetry's signable JSON (what crypto::verify_agent_signature checks) does not include this field, so adding a judge evaluation never requires a client to re-sign, and no existing client's signature breaks. It rides along as an unauthenticated sidecar on an otherwise-authenticated request. No judge/rubric implementation exists anywhere in this codebase — this is plumbing only, per the task scope (a Xibalba Solutions product decision not yet made).

Read-only via alloy (stored as DynProvider so sol! bindings work): resolves an agent's PrimitiveSet, reads ReputationRegistry.effectiveScore/isZkBoosted, ComplianceGate.vertical/isHealthcareCompliant, MarketFactory enumeration, per-market IntegrityMarket view state, getPosition, IntegrityToken.balanceOf, SmartBAAFactory/SmartBAA (Integrity Health), and — new — XibalbaNameService (resolve_handle/primary_handle) and IntegrityGovernance (read_proposals). Optional singleton addresses (MarketFactory, IntegrityToken, A2ACapitalPool, SmartBAAFactory, XibalbaNameService, IntegrityGovernance) are Option<Address> in the deployments-file parse (Singletons), not required: a deployments file that predates any of them still parses, and handlers needing an absent one return a clean ChainError::MissingSingleton — mapped to HTTP 400 in error.rs (a deployment-shape fact, not a transient RPC failure), shared across the market/health/XNS/governance endpoints — instead of this client failing to even connect.

Telemetry leaves batched into a keccak256 Merkle tree; because StateAnchor is per-agent, the same epoch root is submitted to each participating agent's own StateAnchor clone — a documented gas tradeoff.

crypto::canonical_json_bytes, the byte representation checked for signed telemetry, uses serde_jcs. Python producers use the jcs package. This shared RFC 8785 rule replaces the earlier custom ASCII formatter and pins both ECMAScript number serialization and raw non-ASCII UTF-8 across runtimes.

GET /v1/agents/snapshot is the directory interface consumed by Cortex. Its response includes chain_id, block_number, finalized_block_number, finalized_block_hash, snapshot_id, and finalized. The finalized value is the conjunction of the operator approval setting AGENT_DIRECTORY_FINALIZED and the execution client's finalized cursor covering the directory cursor. When approval is enabled, the handler anchors block_number to the execution client's finalized block before evaluating the conjunction; it does not label the moving latest head as finalized.

The Cortex sync worker refuses any snapshot whose finalized value is false, whose finalized cursor is missing or malformed, or whose finalized cursor does not cover the directory cursor. The setting must therefore remain disabled until registration state has been independently checked at a finalized block. On 2026-09-14, read-only Base Sepolia historical calls at block 46830231 resolved all five directory agents with exists=true; the Oracle was then restarted with AGENT_DIRECTORY_FINALIZED=true and returned a finalized snapshot at that same block with five agents.

80 backend + scoring-core lib tests (confirmed via a real run — 72 backend, 8 scoring-core; up from 54 with the derive.rs re-derivation module's own parity-with-derive.py unit tests among the additions), including src/phi.rs unit tests covering every PHI category, the already-redacted-marker non-reflag case, and the numeric-vs-string-field scan boundary), plus a real full-stack e2e (tests/e2e.rs, 9 tests, opt-in via ORACLE_E2E=1) that stands up live anvil + Deploy.s.sol (which now deploys the market layer as part of genesis) + a real SDK-registered agent + Postgres

• Redis + the HTTP server, and asserts accept-correct / reject-fabricated primitives, AIS scoring, the live compliance read, a real (empty, pre-any-market) GET /v1/markets, a real one-entry GET /v1/leaderboard, a real GET /v1/agent/{id}/wallet balance read, the PHI backstop's real HTTP-level 400, and — new — oracle_e2e_recomputed_grounding_overrides_inflated_client_claim, proving an agent that claims an inflated grounding score while its own signed otel_spans contain hallucination markers gets the oracle's real, low recomputation stored and scored, never the client's claim. Documented follow-up, not built this pass: a full market lifecycle e2e (enterPosition/resolve/claimPayout through a second real registered agent) — out of scope for this task; the light e2e above proves the real binding/parse/handler path without that heavier setup.

Related: Telemetry Ingestion Pipeline, AIS, agent primitives, Interface Contract.