07 ‐ Data Ingestion, Monitoring & Alerts - IMGitH/dvmax GitHub Wiki

Ingestion Tasks

• Fetch weekly price data from FMP API and store in /prices/
• Fetch quarterly fundamentals and dividends (same API)
• Normalize and aggregate using Polars, store in Parquet
• Maintain log or metadata with last pull timestamp

Monitoring Jobs

• Weekly: check for missing tickers, NaNs, or drift in distributions
• Raise alerts (email/Slack/log) on:
  • Sudden dividend drops
  • Yield spikes > threshold (possible dividend traps)
  • Missing data for tracked stocks

Tools

• Polars, PyArrow
• Optional: Prefect or Airflow for orchestration
• Notification via email or webhook