Data collectoion API keys - ram290476/investor-dashboard GitHub Wiki
Provider credentials live in SSM Parameter Store as SecureStrings under /invdash/api-keys/<name>. Terraform only creates placeholders (REPLACE_ME); real values are never committed to Git, GitHub secrets, or the static site.
Do not paste key values into chat, issues, or PRs. Use infra/terraform/scripts/rotate-key.sh on a machine signed in with AWS profile invdash-admin (region us-west-1).
| Path | Keys required? |
|---|---|
Yahoo historical backfill → dashboard-build → charts |
No — public Yahoo data |
daily-prices / options-daily
|
Alpaca key id + secret |
short-interest |
FINRA client id + secret |
| Future macro / news / regulatory collectors | FRED, BLS, BEA, Census, Finnhub, Alpha Vantage, Massive, SAM.gov, api.data.gov, ACLED |
Until placeholders are replaced, Alpaca and FINRA jobs will fail; the Yahoo backfill path still works.
| SSM key | Provider | Signup / console |
|---|---|---|
alpaca-key-id |
Alpaca (use paper keys only) | app.alpaca.markets |
alpaca-secret-key |
Alpaca (rotate with key id) | app.alpaca.markets |
finra-client-id |
FINRA Query API (free Public OAuth) | gateway.finra.org |
finra-client-secret |
FINRA (with client id) | gateway.finra.org |
| SSM key | Provider | Signup / console | Notes |
|---|---|---|---|
fred |
FRED | fredaccount.stlouisfed.org/apikeys | Key does not expire; rotate yearly by policy |
bls |
BLS | data.bls.gov/registrationEngine | Renew at least yearly |
bea |
BEA | apps.bea.gov/API/signup | |
census |
Census | api.census.gov/data/key_signup.html | |
api-data-gov |
api.data.gov (FCC) | api.data.gov/signup | |
finnhub |
Finnhub | finnhub.io/dashboard | |
alpha-vantage |
Alpha Vantage | alphavantage.co/support | Free tier is rate-limited (~25 calls/day) |
massive |
Massive (Polygon) | massive.com/dashboard | |
sam-gov |
SAM.gov | sam.gov/profile/details | Personal keys expire every 90 days |
acled-password |
ACLED (myACLED login) | acleddata.com | App exchanges password for short-lived tokens |
From a clone of ram290476/investor-dashboard, with AWS SSO ready:
export AWS_PROFILE=invdash-admin AWS_DEFAULT_REGION=us-west-1
cd infra/terraform
# Minimum for weekday closes + short interest:
./scripts/rotate-key.sh alpaca-key-id
./scripts/rotate-key.sh alpaca-secret-key
./scripts/rotate-key.sh finra-client-id
./scripts/rotate-key.sh finra-client-secret
# Remaining as you create them:
./scripts/rotate-key.sh fred
./scripts/rotate-key.sh bls
./scripts/rotate-key.sh bea
./scripts/rotate-key.sh census
./scripts/rotate-key.sh api-data-gov
./scripts/rotate-key.sh finnhub
./scripts/rotate-key.sh alpha-vantage
./scripts/rotate-key.sh massive
./scripts/rotate-key.sh sam-gov
./scripts/rotate-key.sh acled-passwordEach run prompts silently (value never hits shell history), stores an encrypted SecureString, and triggers an expected API key changed security email.
Optional smoke after Alpaca is set:
aws lambda invoke --function-name invdash-daily-prices --payload '{}' /tmp/daily-prices.jsonA daily checker emails the security topic before each key’s due date (14 / 7 / 3 / 1 / 0 days, then daily when overdue). Periods: SAM.gov 90 days, Alpaca 180 days, others 365 days. Source of truth for “last rotated” is the SSM parameter’s last-modified time.
Canonical table in-repo: docs/architecture-infrastructure.md (API key rotation) and infra/terraform/variables.tf (var.api_keys).
See also: Data collectors — job overview, lake layout, operations and known gaps.