API Architecture - fedjo/CEI-InOE GitHub Wiki
CEI-InOE API Architecture
Overview
The API service provides RESTful access to environmental, energy, dairy production, and solar data collected by the CEI-InOE platform. It serves as the primary interface for external applications, dashboards, and integrations.
Architecture
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β CEI-InOE Platform β
β β
β ββββββββββββ ββββββββββββ ββββββββββββββββ ββββββββββββββββββ β
β β Tago.io β β Airbeld β β FusionSolar β β File Sources β β
β β (energy) β β (weather)β β (solar PV) β β (CSV / XLSX) β β
β ββββββ¬ββββββ ββββββ¬ββββββ ββββββββ¬ββββββββ βββββββββ¬βββββββββ β
β βββββββββββββββ΄ββββββββββββββββ΄βββββββββββββββββββ β
β βΌ β
β βββββββββββββββββββ β
β β Ingestor β β
β β (Scheduled) β β
β ββββββββββ¬βββββββββ β
β β writes β
β βΌ β
β ββββββββββββββββ βββββββββββββββββββ ββββββββββββββββ β
β β Alembic βββββββΆβ PostgreSQL ββββββββ Grafana β β
β β (migrations) β β β β (dashboards) β β
β ββββββββββββββββ β - Fact tables β ββββββββββββββββ β
β β - Staging β β
β β - Metadata β β
β ββββββββββ¬βββββββββ β
β β reads β
β βΌ β
β βββββββββββββββββββ ββββββββββββββββ β
β β API ββββββββ Frontend / β β
β β (FastAPI) β β Dashboards β β
β βββββββββββββββββββ ββββββββββββββββ β
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
Directory Structure
api/
βββ Dockerfile
βββ requirements.txt
βββ app/
β βββ main.py # FastAPI app, router registration, lifespan
β βββ config.py # Pydantic Settings (env vars)
β βββ auth.py # X-API-Key verification dependency
β β
β βββ db/
β β βββ session.py # SQLAlchemy engine + session factory
β β βββ queries/
β β βββ energy.py # Energy data queries
β β βββ environmental.py
β β βββ dairy.py
β β βββ datasources.py
β β βββ batches.py
β β
β βββ routers/
β βββ health.py # GET /health, GET /ready (public)
β βββ energy.py # /api/v1/energy/*
β βββ environmental.py # /api/v1/environmental/*
β βββ dairy.py # /api/v1/dairy/*
β βββ datasources.py # /api/v1/datasources/*
β βββ batches.py # /api/v1/batches/*
Note: Response schemas are defined in
shared/src/shared/schemas.pyand imported by both the API and ingestor. The API has no localschemas/directory.
Authentication
All data endpoints (/api/v1/*) require an X-API-Key header. Health endpoints (/health, /ready) are public.
X-API-Key: <value of API_KEY env var>
The key is validated in api/app/auth.py via a FastAPI dependency injected at the router level in main.py. Missing or mismatched keys return 403 Forbidden.
Design Principles
| Principle | Implementation |
|---|---|
| Database as Contract | PostgreSQL schema (via Alembic migrations) is the source of truth |
| Read-Only Access | API only reads data; ingestor handles all writes |
| SQLAlchemy ORM Queries | Query modules use SQLAlchemy ORM (Session) for type-safe queries |
| Shared Schemas | Pydantic v2 models in shared/ are used for both API responses and ingestor validation |
| Stateless | No session state; scales horizontally |
Data Flow
Request Lifecycle
HTTP Request
β
βΌ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β Router β
β - Validate query parameters (Pydantic / FastAPI) β
β - Verify X-API-Key (dependency) β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β
βΌ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β Query Function (db/queries/*.py) β
β - Build SQLAlchemy ORM query β
β - Execute against session (connection pool) β
β - Return ORM objects or scalar results β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β
βΌ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β PostgreSQL β
β - Query fact / metadata tables β
β - Return result set β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β
βΌ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β Schema Serialisation β
β - Pydantic model validates and serialises response β
β - model_validate(orm_obj) converts ORM β schema β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β
βΌ
HTTP Response (JSON)
Service Interaction
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β Schema Change Workflow β
β β
β Developer β
β β β
β βΌ β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β 1. Update shared/src/shared/models.py β β
β β Add/change SQLAlchemy model β β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β β
β βΌ β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β 2. Generate Alembic Migration β β
β β alembic revision --autogenerate -m "description" β β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β β
β βΌ β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β 3. Update Affected Services β β
β β - shared/src/shared/schemas.py (Pydantic schemas) β β
β β - api/app/db/queries/*.py (query changes) β β
β β - ingestor/app/models.py (validation models) β β
β β - ingestor/mappings/*.yaml (field mapping) β β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β β
β βΌ β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β 4. Test & Deploy β β
β β a. alembic upgrade head β β
β β b. Deploy API (read-safe) β β
β β c. Deploy Ingestor (write-safe) β β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
Related Documentation
- API Reference - Complete endpoint documentation
- Schema Change Guide - How to handle schema changes
- Ingestor Architecture - Data ingestion pipeline