How the investor-dashboard data jobs collect, store and publish data. Sources: services/data-jobs/, infra/terraform/variables.tf (var.jobs, var.api_keys), services/data-jobs/src/app/status_feed.py, docs/operations.md, and issue #41.

Credentials: see Data collectors API keys. This page lists SSM key names only and never their values.

• Runtime: every job is a container-image Lambda named invdash-<job>, built from services/data-jobs. Handlers live in src/functions/<name>/. Terraform var.jobs gives each job its handler, schedule, triggers, memory/timeout, read_prefixes/write_prefixes, api_keys and reads_prefs. The jobs module (infra/terraform/modules/jobs) turns these into Lambdas, IAM and EventBridge rules.
• Schedules: EventBridge Scheduler cron expressions in America/New_York. Jobs check the NYSE / US federal calendars and skip holidays.
• Triggers: event-driven jobs subscribe to job:<ID> patterns on the default bus. ticker-added (from prefs-api) starts the backfill.
• job_handler(job_id) (src/app/observability.py) wraps each handler. It:
  • sets correlation ids (run_id = <JOB>-<UTC ts>-<hex>), X-Ray annotations and metric flushing;
  • records one run per source through the source_run("DS-xx") bulkhead (a failure is recorded and the job moves on to the next source);
  • writes ingestion_runs records;
  • publishes a Job Finished event with {job, run_id, outcome, failed_sources}.
• status-feed (trigger job:*) merges each event into serving/status.json with conditional writes (If-Match ETag). JOBS in status_feed.py lists 16 job ids with display names, planned crons and calendars. A job that has never reported shows status: never_run and last_run: null.
• Data freshness panel (apps/web/app.js → renderStatus()) shows one row per status.jobs[] entry and formatTime(job.last_run). A null last_run displays as "not yet collected".

flowchart LR
  S[EventBridge Scheduler<br/>ET crons] --> C[Collector Lambdas<br/>invdash-*]
  P[prefs-api<br/>ticker-added] --> B[invdash-backfill]
  C -->|api_key from SSM<br/>/invdash/api-keys/*| K[(SSM SecureString)]
  C --> R[(raw/)]
  C --> CU[(curated/ Parquet)]
  B --> CU
  C -- Job Finished --> EB{{default event bus}}
  B -- Job Finished --> EB
  EB -->|job:D4, M1, RECONCILE| T[trend-metrics]
  T --> SV1[(serving/trend_metrics/)]
  T -- Job Finished --> EB
  EB -->|job:D4, TREND, Q1, SHORT,<br/>OPTIONS, BACKFILL, RECONCILE| D[dashboard-build]
  EB -->|job:*| SF[status-feed]
  SF --> ST[(serving/status.json)]
  D --> DJ[(serving/dashboard.json)]
  ST --> D
  DJ --> W[Web app / Data freshness panel]
  ST --> W

Bucket invdash-lake-308639168050 (us-west-1). SSE-KMS with the project data key is the bucket default. Helpers are in src/app/lake.py.

• ingestion_runs: one row per source run (job, run_id, source, outcome, rows, duration), written to curated/ingestion_runs/year=<Y>/month=<M>/<run_id>.parquet.
• Least privilege: each job's IAM role can only write its write_prefixes and read its read_prefixes. bucket and bucket/* are granted separately.
• Per-job folder isolation: a job writes only under its own keys, so concurrent jobs never overwrite each other. The exceptions are the shared price partitions and status.json, both of which use conditional writes.
• Legacy layouts: date=<D>/daily.parquet and batch_start=/batch_end= price objects are still read and deduped. scripts/compact_price_partitions.py folds them into the yearly objects.

All schedules are ET (America/New_York) unless noted. The SSM names are under /invdash/api-keys/.

The prefs-api Lambda serves user preferences (DynamoDB) and emits ticker-added.

Update, Oct 10, 2026: all five collectors below were built and merged on Oct 7 (PRs #42, #43 and #47), and D1 followed in #55. The table is kept as the original design record. See section 7 for what happened when they went live.

These ids are in status_feed.JOBS, but no Lambda or terraform job exists for them yet, so the panel shows "not yet collected". Full design: issue #41.

Each new collector uses @job_handler("<ID>"), a new var.jobs entry, its own write_prefixes, and a dashboard-build trigger and read prefix. Suggested build order: stopgap "not set up yet" label → H1 → H3 → D5 → H2 → M1. M1 also needs a release calendar, and D1 (Morning macro & calendars) is unbuilt too.

All commands assume export AWS_PROFILE=invdash-admin AWS_REGION=us-west-1. If the SSO session has expired, run aws sso login --profile invdash-admin first.

Manually invoke a job

aws lambda invoke --function-name invdash-daily-prices --payload '{}' /tmp/daily-prices.json && cat /tmp/daily-prices.json
aws logs tail /aws/lambda/invdash-daily-prices --since 1h

A manual run emits Job Finished just like a scheduled one, so status-feed and the downstream jobs (trend-metrics, dashboard-build) also run.

Backfill (base tickers plus ETF proxies from universe.py):

aws lambda invoke --function-name invdash-backfill --cli-binary-format raw-in-base64-out \
  --payload '{"tickers":["TSLA","SPCX","SPY","DIA","QQQ","IWM","XLY","ITA","SMH"]}' /tmp/backfill-result.json
aws s3 cp s3://invdash-lake-308639168050/curated/prices_daily/_backfill/state.json -
aws logs tail /aws/lambda/invdash-backfill --follow

The backfill is safe to re-run because it resumes from the checkpoint. Changing backfill_years while a load is active is rejected.

Rotating a key

AWS_PROFILE=invdash-admin infra/terraform/scripts/rotate-key.sh sam-gov

• The script reads the value silently and sends it on stdin, so it never appears in shell history or the process list. It writes /invdash/api-keys/<name> as a SecureString with the data KMS key. An unknown name lists the valid names.
• The parameter's last-modified date counts as "last rotated". The api-key-changed alert emails the security topic.
• Caching: Lambdas cache keys for about 5 minutes (app/api_keys.py). Wait about 5 minutes after rotating before re-invoking, or the old value may still be used.
• Provider rules: sam-gov expires every 90 days, bls must be renewed yearly, and Alpaca keys should be rotated every 180 days. See Data collectors API keys.

Alerts and monitoring

• SNS ops and security topics in the primary region, plus the CloudFront alarm topic in us-east-1.
• CloudWatch alarms and dashboards from the Terraform outputs.
• After two retries, a failed Lambda goes to its dead-letter queue.
• The key-rotation checker sends reminder emails.
• Logs are structured Powertools JSON lines. Filter on job, run_id and event: job_run.

• Short interest isn't in serving. short-interest writes curated/short_interest/, but dashboard-build doesn't read that prefix, so dashboard.json has no short interest. Fix: add the read prefix and a tickers[T].short_interest field.
• SPCX pre-listing backfill shows partial. Yahoo has no history for SPCX before it listed, so that ticker's backfill and reconcile sources report a failure and the run is marked partial (amber dot) even though nothing is broken.
• Short-interest error masking. source_run swallows exceptions as a bulkhead. If the FINRA token or query fails, new_dates is never assigned, and the following if new_dates: raises an UnboundLocalError that hides the real FINRA error in logs and status. Fix: initialise new_dates = set() before the with block.
• prefs-api returns 500 when a serving file is missing. It should treat a missing serving object as empty/unavailable. New serving fields must be optional.
• sec_user_agent must be set. It defaults to "", which makes http_client fall back to a placeholder User-Agent. SEC requires a real contact (investor-dashboard <contact email> in tfvars and the TERRAFORM_TFVARS secret) before H3 ships.
• SAM.gov limits. The basic sam-gov key allows 10 calls a day and expires every 90 days. D5 must budget about 3 calls per run and report partial (not crash) when the key has expired.
• Planned jobs look broken. H1/H2/H3/D5/M1 (and D1, D2, D3, W1) appear as "not yet collected" with a next_run. A stopgap "not set up yet" label is proposed in #41.

Added Sat Oct 10, 2026, about 1:30 PM PT. All times are Pacific (PT). The facts come from the repo's issues, pull requests, commits and GitHub Actions runs unless a row says otherwise.

This section is the history of what went wrong after the collectors were first deployed, why, and what fixed it. It runs roughly in date order. Each entry links to the issue or pull request where the work happened, so you can read the full detail there.

What happened. The site and sign-in worked, but the dashboard had no prices or charts. Two things caused it (recorded on Deployment issues):

• No backfill had ever run. The schedules alone don't load history for the starting tickers. The Yahoo backfill (invdash-backfill) has to be invoked once, and it hadn't been after the jobs stack was restored. So serving/dashboard.json didn't exist.
• All 14 provider keys in SSM were still the Terraform placeholder REPLACE_ME. Alpaca, FINRA and every key-based collector would fail until real values went in.

The frontend made this harder to see. When /dashboard returned something it couldn't read, it quietly showed "nothing published yet" instead of an error.

Fix.

• The backfill was invoked once with the ticker payload. It needs no API keys, because Yahoo is public. The command is in section 5.
• Real keys were installed with rotate-key.sh. Values never go into Git, the wiki or chat.
• PR #40 added proper loading, error and "Try again" states, so a failed or slow read no longer looks like missing data.

What happened. The Data freshness panel showed "not yet collected" for Government contracts, Market prices, News & sentiment, Regulatory & company feeds, and Release-day data. The panel was working correctly. Those five job ids (D5, H1, H2, H3, M1) were listed in status_feed.JOBS, but no Lambda or Terraform job existed for any of them. Re-running jobs or rotating keys could never have fixed this. Issue #41 has the full design for each feed (summarised in section 4).

Fix. Each feed was built on its own branch, all on Oct 7:

Merge conflicts and CI fixes along the way.

• PR #43. Once H1 had merged, all the feed branches touched the same files (dashboard_build.py, variables.tf, apps/web/app.js, the Dockerfile, pyproject.toml). Merging main into the H2 branch left a duplicate datetime import in the middle of dashboard_build.py, and the CI lint step (ruff) failed with 4 errors (run). A follow-up commit repaired the merge, CI went green, and the PR merged.
• PRs #44, #45 and #46. Updating these onto main one at a time concatenated test functions and Terraform job blocks, which produced invalid Python and failed CI on all three branches. Rather than untangle three conflicting branches, #47 started fresh from main and merged the last clean (green) commit of each feed. The three original PRs were closed.

What happened. Even after the five collectors were deployed, H1, H2, H3, D5 and M1 never ran on their schedules. 103 of 104 EventBridge Scheduler target invocations failed in 24 hours. The scheduler role couldn't decrypt the schedules, because it lacked kms:Decrypt on the project data key that encrypts them. Two smaller problems turned up at the same time:

• The release-day job (M1) creates its own one-off schedules, which means passing the invdash-jobs-scheduler role to EventBridge Scheduler. The workload permissions boundary didn't allow that.
• USAspending returned HTTP 422, because the request was missing the required award_type_codes.

Fix. PR #50 (merged Oct 7, 11:17 AM):

• gave the scheduler role kms:Decrypt on the data key;
• added a PassJobsSchedulerRole statement to the boundary, which allows iam:PassRole on invdash-jobs-scheduler only, and only to scheduler.amazonaws.com;
• made an M1 scheduler error count as partial instead of failed;
• fixed the USAspending request (award_type_codes and paging), keeping the watermark when a request fails.

The deploy failure. The first deploy of PR #50, Deploy production run #80 (11:20 AM), failed with:

AccessDenied: invdash-terraform-deploy is not authorized to perform iam:CreatePolicyVersion on invdash-workload-boundary with an explicit deny

This is by design. The CI deploy role is never allowed to change its own permissions boundary, so a boundary change needs an administrator apply (the PR description said so). After the admin apply, the next run, #81 (11:24 AM), succeeded, finding the boundary already up to date. Note that #81 was the successful run, not the failed one.

What happened. The trend-metrics job (#26) could compute z-scores, correlations and net pressure, but it was missing inputs:

• no job collected the daily FRED macro series it expects;
• M1 wrote series ids the trend job didn't recognise, kept only the latest value, and dated each value to the start of its period, which leaks future information into the past.

The full plan is on Implementation plan: trend metrics serving layer (#26).

Fix.

The production check recorded on #26 (Oct 8, about 5:36 PM) found trend-metrics ok, schema v3 live, CPI/core CPI/PCE latest releases present, nine release-window summaries, and core PCE explicitly missing rather than invented. That check confirmed the serving layer, not the health of every provider (see 7.5).

What happened. The CloudWatch review after #26 (Oct 7–8, us-west-1) found that several individual sources fail in production even though their jobs as a whole run. FRED, daily and hourly prices, news, regulatory feeds, fundamentals and options were healthy.

Status. PR #120 is open, not merged (CI green, 310 tests). Issue #69 is open. Merging it doesn't fix production on its own: the affected collectors then have to be re-run within each provider's limits, and row counts checked. A Lambda success doesn't prove the data arrived.

What happened. The 5-year history from the Yahoo backfill is consolidated volume (all US venues). The daily Alpaca bars use Alpaca's free IEX-only feed, about 2–3% of consolidated volume. When Alpaca rows replaced Yahoo rows, volume fell 40–90x overnight for every ticker. For TSLA, 39.5 million shares on Sep 28 became 524,882 on Sep 29. The Volume lane collapsed and any volume-based metric was wrong. Details: #97.

Fix. PR #111 (merged Oct 10, 1:02 AM) kept Alpaca for prices but uses Yahoo for volume:

• price-reconcile now replaces volume for the last 10 sessions with Yahoo's consolidated number, from the chart it already fetches, so there are no extra calls.
• Each bar carries volume_source (DS-05 Yahoo or DS-02 Alpaca), and the IEX print is kept as volume_iex.
• The dashboard serves volume_source_changes so a mixed series is visible.

After the fix, every one of the 10 sessions matched Yahoo exactly for all 7 watchlist tickers. Hourly and intraday volume are still IEX-only.

What happened. Some failures in the site API Lambda (prefs_api) escaped as a bare HTTP 500 with no JSON body and no structured log line: an STS hiccup, an S3 or KMS read error, a truncated dashboard.json, or an EventBridge error after a successful save (#99). The last of these was the worst: a preference that had been saved was reported to the user as a failure.

Fix, part 1. PR #114 (merged Oct 10, 12:58 AM) wrapped the handler in one error boundary:

• Every failure returns JSON {error, code}: STS_UNAVAILABLE, SERVING_READ_FAILED, PREFS_STORE_UNAVAILABLE, SERVING_DOCUMENT_INVALID or INTERNAL, with 503 for transient errors and 502 otherwise.
• A saved PUT whose EventBridge publish fails now returns 200 with backfill_queued: false.
• The web app shows a specific message for each code.
• One CloudWatch metric filter and alarm per code.

The deploy failure. The metric filters used the pattern "event": "<event>" "code": "<CODE>". That isn't valid CloudWatch filter syntax. Deploy run #165 (1:01 AM) and #167 (1:08 AM) both failed with InvalidParameterException: Invalid metric filter pattern. Terraform had already updated the Lambdas but exited with an error, so the static site wasn't published.

Fix, part 2. PR #121 (merged 5:53 AM) switched to JSON filters, { ($.event = "<event>") && ($.code = "<CODE>") }, which match the one-JSON-object-per-line logs. It also added a test so the pattern stays valid. Deploy run #173 (5:57 AM) succeeded. No IAM change was needed: the error was a 400, not AccessDenied.

What happened. The static site is compressed by CloudFront, but /dashboard, /chart/{ticker} and /status come from an API Gateway HTTP API, which doesn't compress. The dashboard sent about 1 MB of JSON on every load (#98). The document also grows with the number of tickers, and a synchronous Lambda response over 6 MB fails outright.

Fix. PR #116 (merged Oct 10, 7:32 AM; deployed by run #178, 7:36–7:39 AM):

• /dashboard, /chart/{ticker} and /status return gzip (Content-Encoding: gzip, Vary: Accept-Encoding) when the client accepts it and the body is over 1 KB. Other clients still get plain JSON.
• dashboard-build writes pre-compressed serving/dashboard.json.gz and serving/chart_data/<T>.json.gz, and the Lambda caches gzip bytes by S3 ETag.
• Response-size guard: if the response would exceed 6 MiB after base64, the API returns 503 RESPONSE_TOO_LARGE instead of a payload API Gateway would reject.
• New metrics DashboardRawBytes and DashboardGzipBytes, and the alarm invdash-dashboard-payload-size fires above 4 MiB raw.

Live check, Oct 10, 2026:

All three responses carried Content-Encoding: gzip. These live numbers come from a manual check on Oct 10 and aren't recorded in the PR or issue. The PR's own test used a smaller synthetic document. Note that the live dashboard (about 3 MB raw) is already well above the 1 MB in the Oct 7 snapshot, which is why the size guard and alarm matter.

What happened. Deploy production runs on workflow_run after CI. CI also runs on pull-request branches, so every PR push started a Deploy run that immediately skipped (for example runs #168–#172 and #174). That cluttered the Actions history and made real deploy failures harder to spot.

Fix. PR #124 (merged Oct 10, 6:45 AM) added branches: [main] to the trigger. Deploys still happen for pushes to main and for manual runs.

What happened. The per-stock company page (/#stock/TSLA, from #87) always said "No company-specific metrics discovered". The page, API route and IAM grant were live. What was missing was the data (#126):

• The company_ir collector was a placeholder. #87 shipped the page and a helper library, but the three job handlers were stubs (collect only planned, extract returned 0, serve returned 0).
• Every TSLA catalog metric was proposed, so nothing would have shown even with data.
• The manual fundamentals CSV had no deliveries or FSD subscriber rows. Q1 treated a missing CSV as "no rows" without a warning. The repo's example command also used 497,099 as Q3 2026 deliveries; that's Q3 2025. Q3 2026 was 486,532.

Fix. PR #132 (merged Oct 10, 10:01 AM; deployed by run #182, which started at 10:05 AM) built the collector on SEC EDGAR:

• Sources: submissions JSON, the 8-K Item 2.02 EX-99.1 exhibits (the deliveries press release and the quarterly update deck), the latest 10-Q/10-K, and XBRL companyfacts.
• Tesla's websites aren't scraped. www.tesla.com and the ir.tesla.com HTML pages return 403 to automated requests, and the collector doesn't work around that. A deck PDF on ir.tesla.com is a fallback only after checking robots.txt, and a 403 or 429 stops that host.
• Discovery and approval registry: a new row label in a known table becomes a proposed entry in catalog/<TICKER>.json. Nothing is auto-approved. Approval goes through scripts/approve_metric.py. The TSLA catalog now has 47 entries, all proposed.
• Dedupe priority: key (ticker, metric_id, fiscal_period). XBRL wins for GAAP items. For operating items, the latest deck beats the press release. News never wins. A changed later value is kept as a revision (supersedes). Values about 100x off their neighbours are flagged as possible scale errors.
• Once approved, deliveries and FSD subscriptions replace the manual CSV rows in Q1.

More detail is on Company Metrics Collector.

Two on-demand runs were made on Saturday Oct 10, after the deploy. The run details come from the Company Metrics Collector page and the run review. PR #135 confirms the problems found.

Run 1, scheduler payload ({"source":"schedule"}), about 12:04 PM: skipped. This is expected. plan_collection collects on a schedule only on Mondays (UTC date) or inside an earnings window (3 days before to 5 days after an earnings date). The scheduler payload carries no earnings dates and Oct 10 is a Saturday, so collect skipped before contacting EDGAR. Extract and serve skipped with "upstream skipped". No errors.

Run 2, weekly payload ({"reason":"weekly"}), about 12:06 PM: partial.

Fix. PR #135 (CI green; merged Oct 10, 12:23 PM; deployed by run #183, 12:27–12:30 PM):

1. The 8-K index URL is now index.json (verified 200 against a real Tesla filing).
2. Collect publishes partial or failure to match the run. Extract and serve still run after a partial collect, but not after a failed one.
3. Cash is now CashAndCashEquivalentsAtCarryingValue plus ShortTermInvestments (or MarketableSecuritiesCurrent) per quarter. That gives $43.524B for 2026Q2.

The scale-error check was deliberately left unchanged.

The Data refresh panel shows the next Company IR collect run as Tue Oct 13. The status feed's entry for Q2C uses the federal holiday calendar, so it skips Mon Oct 12 (Columbus Day). EventBridge Scheduler doesn't know about holidays. The real schedule, cron(15 7 ? * MON-FRI *) in America/New_York, fires on Mon Oct 12 at 4:15 AM PT. That's a Monday in UTC too, so the weekly collect will run. The panel is wrong; the job isn't. This hasn't been fixed yet.

• Approve TSLA company metrics. All 47 catalog entries are proposed, so the company page shows nothing. Approve with services/data-jobs/scripts/approve_metric.py. An approval UI is tracked in #137.
• Confirm the PR #135 fixes in a real run. Nothing has been collected since the 12:30 PM deploy. The first scheduled weekly collect is Mon Oct 12, 4:15 AM PT. Check that 8-K exhibits are found, that deliveries and FSD rows appear, that the outcome is partial/failure when appropriate, and that cash runs through 2026Q2.
• Fix the status feed's next-run date for Q2C so it matches the scheduler (7.12).
• Review and merge PR #120 for the #69 provider failures. Then re-run the affected collectors within provider limits and check row counts.
• Review the 16 flagged scale-error values (TSLA revenue_gaap 2010–2012, eps_diluted, deferred_revenue).
• Watch the dashboard payload size. The live raw /dashboard is about 3 MB. The alarm fires at 4 MiB, and the hard limit is 6 MiB. Splitting dashboard.json per ticker is the longer-term fix suggested in #98.
• Company page follow-up work (operating-KPI panels, empty states, theme parity) is tracked in #137. The per-stock vision is in #87. How the page works today: Company-specific ticker page.

• Implementation plan: trend metrics serving layer (#26) — daily FRED macro (D1), M1 release-series fix, trend-metrics refactor, triggers and schema v3.
• Company-specific ticker page — the per-stock page that serving/stock/<T>.json feeds: panels, proposed vs approved, and the metric approval how-to.