Architecture Decision Records - antholim/Pulse GitHub Wiki

Architecture Decision Records (ADRs) capture the significant technical decisions behind PULSE: the context, the options considered, what was chosen and what it costs. A decision is never edited after it is accepted. If it changes, a new ADR supersedes it and the old one is marked Superseded by Dn.

Ticket: #15. Most decisions were made with Omerlo's CTO at the technical sync of 2026-10-14.

Index

# Decision Choice Status Date
D1 Hosting Fly.io app plus managed Postgres Accepted 2026-10-14
D2 Pipeline language Python Accepted 2026-10-14
D3 Search Deferred to Iteration 5 or later; then Meilisearch or Typesense rebuilt from Postgres Accepted 2026-10-14
D4 Storage Postgres plus a raw payload archive Accepted 2026-10-14
D5 Tenancy tenant_id on every table, enforced in the query layer, proven by a CI test Accepted 2026-10-14
D6 Ingestion Scheduled pull with an updated_at watermark, behind a webhook-ready interface Accepted 2026-10-14
D7 Deployment shape Standalone service that reads Omerlo through its APIs Accepted 2026-10-14
D8 Omerlo access Backoffice API with an API key in x-api-key, read only Accepted 2026-10-19

Libraries chosen under these decisions are listed in Infrastructure and Tools.

D1. Hosting

  • Status: Accepted, 2026-10-14
  • Context: Release 1 (Nov 17, 2026) must be a hosted deployment. PULSE needs one web app, one scheduled job and one Postgres database, run by a student team with little ops time. Omerlo is moving off AWS.
  • Options considered:
    • AWS (ECS or Elastic Beanstalk, RDS): the most flexible, but needs the most setup (VPC, IAM, load balancer). It is also the platform Omerlo is leaving.
    • Fly.io (app plus managed Postgres): deploys a Docker image in a few commands, has managed Postgres and built-in secrets, and is cheap at our scale.
    • Render or Railway: similar simplicity, but Omerlo did not suggest them.
  • Decision: Fly.io app plus Fly's managed Postgres.
  • Consequences:
    • Deployment stays close to our existing Dockerfiles.
    • Secrets live in Fly.io secrets.
    • Hosting cost must be confirmed with the sponsor.
    • The scheduled sync and the raw archive storage need a Fly-compatible setup, investigated in the Fly.io spike (#64).

D2. Pipeline language

  • Status: Accepted, 2026-10-14
  • Context: The pipeline (connectors, normalization, entity resolution, change detection) is the core of the project. Omerlo itself is written in Elixir.
  • Options considered:
    • Elixir: matches Omerlo's stack and is good at concurrent ingestion, but nobody on the team knows it.
    • Python: the team knows it, and the ecosystem fits the work: FastAPI, SQLAlchemy, rapidfuzz for fuzzy matching, data tooling for the evaluation.
  • Decision: Python for the backend and the pipeline (FastAPI, SQLAlchemy).
  • Consequences:
    • One language across the API and the pipeline.
    • Omerlo's team cannot reuse PULSE code directly, which is acceptable because PULSE is a standalone service (D7).

D3. Search

  • Status: Accepted, 2026-10-14
  • Context: Search and filters are a SHOULD. Release 1 volumes (hundreds to a few thousand organizations per tenant) are small.
  • Options considered:
    • Elasticsearch now: powerful, but heavy to run, and every write has to go to two stores.
    • Postgres queries only: enough for Release 1.
    • Meilisearch or Typesense later: light, typo-tolerant, simple to host.
  • Decision: No search engine before Iteration 5. When it is added, use Meilisearch or Typesense, with the index rebuilt from Postgres by a job, never dual-written by application code.
  • Consequences:
    • Release 1 lists use Postgres filtering and pagination.
    • Postgres stays the only source of truth, so the search index can always be thrown away and rebuilt.

D4. Storage

  • Status: Accepted, 2026-10-14
  • Context: PULSE must keep history, reprocess data when rules change, and trace every figure to its source.
  • Options considered:
    • Data warehouse or lakehouse (Databricks, Snowflake): built for volumes we will not have, at a cost and complexity we cannot justify.
    • Postgres only, with payloads in JSON columns: simple, but bloats the main tables.
    • Postgres plus a raw payload archive (object storage, or a local folder in development).
  • Decision: Postgres is the single source of truth. Raw payloads are archived before any transformation, keyed by tenant, sync run and entity kind, and Postgres stores a pointer and a hash.
  • Consequences:
    • Any record can be reprocessed from its raw payload.
    • Whether Omerlo allows archiving is being confirmed in #23.
    • Archive retention must be covered by the Law 25 page (#34).

D5. Tenancy

  • Status: Accepted, 2026-10-14
  • Context: PULSE will serve several Omerlo clients. One client's data must never reach another, which is also the Law 25 boundary.
  • Options considered:
    • One database per tenant: the strongest isolation, but costly to run and migrate at our stage.
    • Postgres row-level security: strong, but harder to test and debug for the team.
    • tenant_id on every table, enforced in the query layer: cheap now, and expensive to add later.
  • Decision: tenant_id on every table from the first migration, enforced by a tenant-scoped data access layer, and proven by a cross-tenant isolation test in CI (#59).
  • Consequences:
    • Every query goes through the data access layer.
    • One database per tenant stays an upgrade path if a client requires it.

D6. Ingestion

  • Status: Accepted, 2026-10-14
  • Context: Omerlo has no webhooks today and no rate limits, and results can be ordered by updated_at. Whether an "updated since" filter exists is being confirmed in #23.
  • Options considered:
    • Webhooks only: not available.
    • Full reload on every sync: simple, but wasteful as data grows.
    • Scheduled pull with a watermark: works today, and can accept webhooks later.
  • Decision:
    • Scheduled pull with an updated_at watermark per source.
    • A periodic full sync to detect deletions.
    • Every connector implements one interface (for example fetch_since(watermark) returning raw records), so file imports, other sources or webhooks plug in later.
  • Consequences:
    • Each sync run is a snapshot that change detection compares.
    • Removals are only seen by full syncs.

D7. Deployment shape

  • Status: Accepted, 2026-10-14
  • Context: PULSE may be sold as an add-on for Omerlo clients. That could mean code inside Omerlo, or a separate service.
  • Options considered:
    • Plugin inside Omerlo: tight integration, but it ties PULSE to Omerlo's codebase, stack (Elixir) and release cycle.
    • Standalone service: has its own database, authentication and UI, and reads Omerlo through its public APIs.
  • Decision: PULSE is a standalone service that reads Omerlo through its APIs. "Omerlo add-on" is how it is packaged and sold, not how the code is built.
  • Consequences:
    • PULSE can read other sources besides Omerlo.
    • It needs its own authentication and user management (#10).
    • It depends on what Omerlo's APIs expose.

D8. Omerlo access through the backoffice API

  • Status: Accepted, 2026-10-19
  • Context: At the 2026-10-14 sync, PULSE was to read Omerlo through the Reader API with OAuth 2 client credentials and an x-omerlo-media-id header, and the context of D6 and the first connector tickets were written for it. Omerlo has since given the team access to its private backoffice API instead. Its key differences are listed on the Omerlo API page:
    • Authentication is an API key in the x-api-key header. The team creates the key itself from an integration account once Omerlo makes us admins on staging; nobody sends us credentials.
    • Community data is scoped by workspace, and list endpoints return a summary only. Contact, address, description and categories come from a detail endpoint, and metadata from a separate /metadata endpoint.
    • Lists can be sorted by updated_at but not filtered by it.
    • The key can also create, update and delete records.
    • Rate limits are not documented. The "no rate limits" answer of 2026-10-14 was about the Reader API.
  • Options considered:
    • Wait for Reader API credentials: matches the existing tickets, but nothing has been issued and the connector work would stall.
    • Backoffice API with an API key: available now and exposes the community data PULSE needs, but it is a private API with a key that can write.
  • Decision:
    • PULSE reads Omerlo through the backoffice API, sending the key in the x-api-key header. The api_key query parameter is never used, because URLs end up in logs.
    • The connector only calls read endpoints, and the team asks Omerlo for a read-only integration account.
    • One key per environment: staging keys in local .env files and the private credentials repository, production keys in Fly.io secrets.
    • If Omerlo later grants Reader API access, a new ADR decides whether to switch.
  • Consequences:
    • Authentication becomes a simple header instead of an OAuth token flow (#53).
    • Syncing one organization or person takes up to three calls (list, detail, metadata), and fetchers run per workspace (#54).
    • D6 still holds. Without an "updated since" filter, an incremental sync reads lists newest first and stops at the first record older than the watermark, and the watermark is kept per source and per workspace (#55).
    • The synthetic fixtures and the mock server must follow the backoffice shapes (#79, #51).
    • A leaked key could change Omerlo data until it is revoked, which makes key handling part of the security baseline (#22, Security).
    • How Omerlo workspaces map to PULSE tenants must be agreed (#23).

Template

Copy this block for a new decision, give it the next number, and add it to the index.

## Dn. Title

- **Status:** Proposed | Accepted | Superseded by Dm, YYYY-MM-DD
- **Context:** What forces this decision: constraints, requirements, facts.
- **Options considered:**
  - **Option A:** pros and cons
  - **Option B:** pros and cons
- **Decision:** What we chose.
- **Consequences:** What becomes easier or harder, and what follow-up work it creates (link issues).