Data Access Boundary - noetl/ai-meta GitHub Wiki

Data Access Boundary

Authoritative rule: agents/rules/data-access-boundary.md ยท Pairs with: agents/rules/execution-model.md

TL;DR

NoETL platform data is accessible via the NoETL server API only. Workers โ€” including the system worker pool โ€” call the server's HTTP API for any read or write to NoETL-owned tables.

NoETL-owned tables: noetl.event, noetl.command, noetl.execution, noetl.outbox, noetl.catalog, noetl.credential, noetl.keychain, noetl.runtime, any future noetl.*.

Visual

   โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”    HTTP     โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
   โ”‚ Rust worker      โ”‚โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ–บโ”‚  NoETL server        โ”‚
   โ”‚ (user pool)      โ”‚             โ”‚  (Python today,      โ”‚
   โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜             โ”‚   Rust eventually)   โ”‚
                                    โ”‚                      โ”‚
   โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”    HTTP     โ”‚  โ–ฒ                   โ”‚
   โ”‚ noetl-worker-    โ”‚โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ–บโ”‚  โ”‚ ONLY this talks   โ”‚
   โ”‚ system-pool      โ”‚             โ”‚  โ”‚ to Postgres       โ”‚
   โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜             โ”‚  โ”‚                   โ”‚
                                    โ””โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
                                       โ”‚
                                       โ”‚ SQL
                                       โ–ผ
                                    โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
                                    โ”‚  Postgres            โ”‚
                                    โ”‚  noetl.event         โ”‚
                                    โ”‚  noetl.command       โ”‚
                                    โ”‚  noetl.outbox        โ”‚
                                    โ”‚  noetl.catalog       โ”‚
                                    โ”‚  noetl.credential    โ”‚
                                    โ”‚  ...                 โ”‚
                                    โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜


                       Exception โ€” external subsystems
                       (auth, credential rotation, alerting)
                                       โ”‚
                                       โ”‚ direct via tool kinds
                                       โ–ผ
                                    โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
                                    โ”‚  External systems    โ”‚
                                    โ”‚  (Auth0, Vault,      โ”‚
                                    โ”‚   PagerDuty, ...)    โ”‚
                                    โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜

Workers calling /api/internal/* carry the system pool's service-account token; user-pool workers can't call those routes (403). Everything else (/api/events, /api/catalog/*, etc.) is open to all worker pools with appropriate auth.

Auth flow detail (kind-validated 2026-06-02)

The bearer-token plumbing reuses only existing NoETL primitives โ€” no new code:

+------------------------------------------+
|  K8s Secret  noetl-internal-api-token    |
|    key=token, value=<32-byte hex>        |
+--------------------+---------------------+
                     |
                     | valueFrom.secretKeyRef
                     v
+------------------------------------------+
|  Pod env (worker-system-pool)            |
|    NOETL_INTERNAL_API_TOKEN=<token>      |
|    NOETL_KEYCHAIN_ENV_VARS=NOETL_INTERNAL_API_TOKEN
+--------------------+---------------------+
                     |
                     | worker startup
                     | load_keychain_env_allowlist()
                     v
+------------------------------------------+
|  ctx.secrets["NOETL_INTERNAL_API_TOKEN"] |
|    = "<token>"                           |
+--------------------+---------------------+
                     |
                     | playbook command dispatch
                     v
+------------------------------------------+
|  tool: http                              |
|    auth:                                 |
|      type: bearer                        |
|      credential: NOETL_INTERNAL_API_TOKEN|
|  AuthResolver.resolve_bearer():          |
|    ctx.get_secret("NOETL_INTERNAL_..")   |
+--------------------+---------------------+
                     |
                     | HTTPS
                     v
+------------------------------------------+
|  /api/internal/* + Authorization header  |
|    server: secrets.compare_digest        |
|  โœ… 200 OK                               |
+------------------------------------------+

Five hops, each an existing primitive: K8s Secret, env mount, NOETL_KEYCHAIN_ENV_VARS allow-list, AuthResolver, server's constant-time comparison gate.

Phase 2.a.3 turned out to be configuration-only โ€” no Rust code change. The deployment manifest adds two env vars; the playbook references the alias via the standard auth: block.

Why

Three load-bearing reasons:

  1. Connection pool isolation โ€” workers scale 1โ†’50+ on backlog; direct DB access blows the platform connection pool and deadlocks the server's own API.
  2. Sharding readiness โ€” server may shard by execution_id; API boundary makes shard routing transparent to workers.
  3. Single point of consistency โ€” schema migrations, audit logging, RBAC, scrub all enforced at the server. Direct workers bypass them.

The exception โ€” external-subsystem playbooks

When a playbook's target is an external system (Auth0, Vault, GCS, PagerDuty, a tenant's Postgres), it uses tool kinds direct. That's just NoETL acting as a client to an external system โ€” the normal playbook pattern.

Examples of external-subsystem system playbooks (exempt from the rule):

  • system/auth โ€” talks to Auth0 / Okta / SAML.
  • system/credential_rotate โ€” talks to GCP Secret Manager / AWS Secrets Manager / Vault.
  • system/notify_alert โ€” talks to PagerDuty / OpsGenie / Slack.

Examples of NoETL-state system playbooks (rule applies):

  • system/outbox_publisher โ€” reads noetl.outbox โ†’ HTTP to /api/internal/outbox/*.
  • system/projector โ€” writes noetl.event โ†’ HTTP to /api/internal/events/project.
  • system/scheduled_cleanup โ€” TTL on noetl.* tables โ†’ HTTP to /api/internal/cleanup/*.

Server-side implications

The Python server's API surface doesn't expose endpoints for the operations the system pool's playbooks need. These get added in Phase 1 of #46.

Initial inventory:

Endpoint Replaces direct DB access by
POST /api/internal/outbox/claim?limit=N Python claim_outbox_batch
POST /api/internal/outbox/mark-published Python mark_outbox_published
POST /api/internal/outbox/mark-failed Python mark_outbox_failed
GET /api/internal/outbox/pending-count (new) KEDA HTTP scaler source
POST /api/internal/events/project Python projector batch INSERT

Auth gate: service-account token only the system pool's K8s ServiceAccount carries. User playbooks calling /api/internal/* get 403.

Decision tree for a new playbook

  1. Does it touch noetl.* tables?
    • Yes โ†’ API only.
    • No โ†’ tool kinds direct.
  2. Does the API have the endpoints needed?
    • Yes โ†’ use them.
    • No โ†’ file sub-issue on noetl/noetl / noetl/server to add the endpoint before writing the playbook.
  3. Auth? Always โ€” /api/internal/* requires the system pool's service account token.

History

Codified 2026-06-02 (afternoon) per standing instruction. Full rationale + endpoint inventory in the authoritative rule above.

โš ๏ธ **GitHub.com Fallback** โš ๏ธ