Issues hit during Terraform apply, GitHub Actions Deploy, Cognito/site cutover, and ops for ram290476/investor-dashboard, and how they were fixed. Companion page: Build issues.

Account / region context: AWS 308639168050, primary us-west-1, deploy role invdash-terraform-deploy, site CloudFront (e.g. https://dbkmugvl0zmt0.cloudfront.net).

Symptom	Deploy failed at configure-aws-credentials with Not authorized to perform sts:AssumeRoleWithWebIdentity. Site still showed Access Denied until this worked.
Cause	GitHub’s OIDC sub for this repo uses immutable IDs (repo:owner@id/repo@id:environment:production), not the legacy repo:owner/name:environment:production form alone. Trust policy only matched one shape.
Remediation	PR #13 updated the role trust to accept both sub formats, plus exact repository_id / repository_owner_id / ref / job_workflow_ref. Compare CloudTrail’s failed AssumeRoleWithWebIdentity userIdentity.userName (sub) to the table in docs/operations.md. Trust changes need an administrator apply (CI cannot PutRolePolicy on itself).

Symptom	Deploy run after PR #14 (e.g. 37496345092) failed mid-run but destroyed ~95 job resources (Lambdas, schedules, IAM). Lake/site remained; collectors gone.
Cause	Workflow “bootstrap” step ran terraform apply -var='jobs_image_uri='. With jobs already in state, empty URI → enabled_jobs = {} → Terraform plans full job destroys.
Remediation	1) Admin restore apply with the last good ECR tag (e.g. …/invdash/jobs:<sha>-1), never an empty URI. 2) PR #15: skip empty-image apply once jobs/ECR exist; refuse apply if JOBS_IMAGE_URI unset; fix SIGPIPE bootstrap check (Build issues). 3) Local admin plans in a worktree must pass the deployed jobs_image_uri and sec_user_agent or they will plan the same 95 destroys — never apply that plan.

Symptom	Deploy apply AccessDenied on iam:PutRolePolicy for invdash-terraform-deploy when expanding PassRole (e.g. Cognito SMS MFA role).
Cause	By design: explicit deny of self-modification on the deploy role / OIDC / boundary so CI cannot privilege-escalate.
Remediation	Apply module.github_deploy changes (trust, iam-for-this-stack PassRole allowlist, boundary) with AdministratorAccess (SSO profile invdash-admin) before expecting CI to succeed. Do not weaken the self-modification deny.

Symptom	Enabling SMS MFA failed with Cognito AccessDeniedException (PassRole to invdash-cognito-sms). Separately, US SMS needs toll-free registration (can take many business days).
Cause	Deploy role lacked PassRole to Cognito until admin allowlist update; SMS sandbox + registration is outside Terraform.
Remediation	Short term: Cognito MFA off (password-only), 15‑minute auth session (AuthSessionValidity), remove SMS role — applied intentionally after user approval; PR #15 encodes MFA-off in Terraform. Identity Center / console MFA for AWS remains required. SMS registration forms were prepared report-only and not submitted unless explicitly asked.

Symptom	Authenticator codes rejected during MFA setup; login flaky.
Cause	Default Cognito auth session window was 3 minutes; stale QR / old authenticator secrets from repeated setup attempts.
Remediation	Raised client auth_session_validity to 15 minutes. Prefer a fresh authenticator entry after any reset. MFA later disabled for the app (see above); AWS SSO MFA unchanged.

Symptom	AWS Config could not deliver to the audit bucket.
Cause	Audit bucket Object Lock / retention blocked Config’s delivery prefix.
Remediation	PR #10: dedicated Config bucket (separate from Object Locked audit storage). Re-verify Config recorder/delivery channel after apply.

Symptom	Plan/apply under PowerUserAccess blocked IAM roles, password policy, Access Analyzer, etc.
Cause	Managed PowerUserAccess excludes IAM (and related) that the stack creates on day one.
Remediation	First full apply used SSO AdministratorAccess (rvellamsetti). Later routine deploys use OIDC invdash-terraform-deploy (PowerUser + bounded IAM). Keep admin only for bootstrap and github_deploy changes.

Symptom	Early plans assumed us-east-1; product decision was primary us-west-1 with us-west-2 DR and us-east-1 only where required (CloudFront / IAM sign-in alarms).
Cause	Repo defaults differed from the chosen deployment region.
Remediation	Infra PR switched defaults; tfvars / GitHub AWS_REGION / TERRAFORM_TFVARS must stay aligned with us-west-1.

Symptom	Cognito password login works; UI shows no data.
Cause	Data lake empty (serving/dashboard.json missing). Yahoo backfill never invoked after job restore; all 14 SSM API keys still REPLACE_ME. Schedules alone do not start the first ticker backfill.
Remediation	Invoke invdash-backfill once with the ticker payload (no API keys required for Yahoo). Then rotate real keys with rotate-key.sh — see Data collectors API keys. Optional smoke: invdash-daily-prices after Alpaca keys are set.

Symptom	Ops/security mail went to an old address; Cognito username is case-sensitive.
Cause	Alert emails updated to [email protected]; Cognito user created as [email protected] (exact casing).
Remediation	Keep tfvars alert list current; sign in with the exact Cognito username string. Confirm SNS topic subscriptions if alerts are missing.

Symptom	CloudWatch alarms firing: site 5xx, prefs-api SERVING_READ_FAILED, Lambda errors/throttles, failed job runs.
Cause	Account Lambda concurrency cap of 10, prefs-api missing s3:ListBucket, FINRA 204 empty bodies, changed release-day sources, LeaseHeld errors (details below).
Remediation	PRs #178–#181 merged and deployed; Lambda quota increase pending; post-deploy checklist in progress.

1. First step: AWS sign-in approval

• The box's admin profile invdash-admin uses IAM Identity Center SSO device login: aws sso login --profile invdash-admin --use-device-code --no-browser.
• Ram must approve the device code at https://d-91677448af.awsapps.com/start/#/device. Codes expire in about 10 minutes.
• Codes CNBM-NKHV and NDQV-DHXT expired unapproved; NVMW-WHSK was approved at about 7:24 PM PT. No verification ran before approval.

2. Root causes (diagnosis Oct 13)

• Account Lambda concurrency cap of 10 with no reserved concurrency: TickerAdded fans out to 5 jobs, plus Job Finished → status-feed/dashboard-build, plus prefs traffic. This throttled prefs-api and caused the site 5xx.
• prefs-api missing s3:ListBucket on serving/stock/*, so a missing object returned 403 and SERVING_READ_FAILED.
• FINRA returning 204 with an empty body, which was parsed as JSON.
• Release-day sources changed: BLS cpi.ics 404, Cleveland Fed layout change, Census 400 from the query.
• LeaseHeld raising errors and retries.
• Ruled out: company_ir, the 8-K 404, KMS.

3. Fixes

• #178: prefs-api s3:ListBucket on serving/stock/*; a missing ticker returns 404 STOCK_NOT_READY; optional prefs_api_reserved_concurrency var (set to 3 after the quota increase).
• #179: short-interest treats 204/empty as no rows, with retry/backoff for non-JSON.
• #180: LeaseHeld logs job_skipped and exits cleanly via the shared job_handler.
• #181: BLS bls.ics feed, Cleveland parser for the transposed tables, Census cell_value/BOPG.
• Ram approved the merges at 6:17 PM PT; merged about 6:18 PM PT. Deploys finished 6:25 (#178), 6:33 (#179), 6:39 (#180) and 6:44 PM PT (#181).
• Note: the queued merges cancelled CI runs, which were re-run one by one.

4. Quota

• AWS Lambda concurrent executions in us-west-1: 10 → 1000. Request 615df9a25e17402e89de1fd03c366355pjT7TyR9 filed at 10:51 AM PT Oct 13; status: approved (CASE_CLOSED at 12:23 PM PT; account concurrency now 1000).

5. Post-deploy checklist (completed Oct 13)

1. Verify the deploy: Lambdas updated, ListBucket present, #180 code live.
   Results (7:24–7:27 PM PT): verified. Five job Lambdas run image 2449022 (#181 merge, includes #178–#181); prefs-api updated 6:25 PM PT with STOCK_NOT_READY; role has ListBucket on catalog/* and serving/stock/*; #180 skip_when_leased=True confirmed in the deployed image.
2. Purge invdash-job-dlq (112 stale Oct 7–12 messages, mostly dashboard-build LeaseHeld plus already-fixed bugs) ONLY after #180 is verified, and only with Ram's explicit go-ahead.
   Results: 118 messages purged at 7:25:57 PM PT; queue at 0 by 7:27 PM PT.
3. Rerun invdash-short-interest and invdash-news-sentiment (#177 all-ticker news check).
   Results: short-interest returned 200 with no new FINRA dates (empty reply handled). news-sentiment returned 200, 102 rows; all 12 watchlist tickers have RSS headlines. Finnhub has returned empty lists since 6:16 PM PT (suspected date-window issue, unconfirmed).
4. Check alarms (invdash-lambda-errors, failed-runs, throttles, site-api-5xx, prefs-api-serving-read-failed) and the quota status.
   Results: all alarms OK except dlq-not-empty (pending re-evaluation). lambda-errors at 3:46 PM PT came from short-interest (pre-#179); 5:41 PM and 6:41 PM PT came from invdash-regulatory-feeds (polars ComputeError on name strings), which is still failing hourly and is NOT fixed by these PRs. Quota: CASE_CLOSED at 12:23 PM PT; account concurrency is now 1000. Reserved concurrency for prefs-api is not set yet.

1. Never apply a plan that destroys the jobs stack because jobs_image_uri is empty or wrong.
2. Never put provider API keys in Git, wiki, chat, or TERRAFORM_TFVARS — only rotate-key.sh → SSM.
3. Never let CI modify invdash-terraform-deploy / OIDC / boundary; use admin apply.
4. Protect GitHub environment production with required reviewers and main only.

• docs/operations.md — deploy role trust table, empty-image rules, Cognito MFA note
• Build issues — CI / workflow / for_each / SIGPIPE
• Data collectors API keys — key signup and install