Mailbox: [email protected] Review run: on demand, Saturday Oct 10 2026, around 12:00 PM PT Window covered: new mail since the previous check at 11:09 AM PT Scope: read-only email review. The AWS account itself was not queried.

Standing note: This report is kept current with new investor-dashboard alert and ops issues as they are filed and fixed, starting with #182–#184. When a related issue is opened or closed (alarm causes, prefs-api serving/chart permissions, collector date windows, notifier behavior), add it to the "Tracked issues" section below with its status.

Issue	Topic	Status
#182	Red alarms: invdash-failed-runs (Census empty body) and invdash-dlq-not-empty (regulatory-feeds polars ComputeError); alert notifier suppresses OK emails (1 per alarm per hour, dedup ignores ALARM/OK)	Closed (Oct 14). Merged: #186 regulatory-feeds schema, #187 Census (BOPGS SA, empty body = no data), #188 notifier dedup by state. Oct 14 ~3:16 PM PT: post-deploy invdash-regulatory-feeds job_finished success hourly since 9:25 AM PT (18/18 sources); invdash-job-dlq was already empty and was purged; invdash-dlq-not-empty OK. invdash-failed-runs still ALARM from gov-contracts D5 (SAM.gov HTTP 400, defense.gov Contracts 403), not regulatory-feeds.
#183	prefs-api chart route AccessDenied / SERVING_READ_FAILED for missing serving/chart_data/ (extends #178)	Closed (Oct 14). Merged #185: s3:ListBucket on serving/chart_data/*, missing chart → 404 CHART_NOT_READY, ticker logged on chart-route errors.
#184	news-sentiment Finnhub empty in the evening (date window); log ticker on empty fetches	Closed (Oct 14). Merged #189: Finnhub window yesterday through today (ET), finnhub_fetch log with ticker and row count.
#216	invdash-failed-runs ALARM from gov-contracts D5: SAM.gov opportunities HTTP 400 (missing postedFrom/postedTo) and defense.gov Contracts 403	Closed (Oct 14). Merged #217; jobs redeployed ~5:10 PM PT. invdash-failed-runs back to OK at 5:46 PM PT. Oct 15 verification (email only): after the 2:45 PM PT gov-contracts window, invdash-failed-runs still flapped ALARM/OK (ALARM about 9:19 AM, 12:19 PM, 2:46 PM, and 4:16 PM PT; OK notices between; latest OK about 5:16 PM PT). Alarm emails do not name the failed sources; CloudWatch job logs needed to tell whether #217 held or another job is hitting FailedRuns ≥ 2.
#221	Security: Amazon Inspector CVEs on invdash/jobs image (2 critical Go stdlib, 17 high)	Closed. Merged #222: rebuilt aws-lambda-rie with Go 1.27.2 + chi v5.3.0, pinned base digest, upgraded AL2023 packages and pip.
#227	invdash-site-api-5xx bursts Oct 15: all GET /fund/{ticker} 500s. Missing Lambda invoke grant for fund/* / admin/fund-flag/* (missed in #224); fund panel also re-fetched in a tight loop on error.	Closed (Oct 15). Fixed by #228 (merged, Deploy production success). Resource policy now has AllowApiGateway-fund and AllowApiGateway-fund_flag_reset. Unauthenticated GET /fund/ARKK → 401 (authorizer). invdash-site-api-5xx OK.
#229	Admin tab fund-flag reset fails with "Load failed" (SYFY, RBI, AGIQ). Cause: the site HTTP API CORS allow_methods was GET/PUT only, so the browser's POST preflight for POST /admin/fund-flag/{ticker}/reset got no CORS headers and the request never reached the Lambda. How the ticker was stamped didn't matter.	Fixed (Oct 15). Merged #230 (POST added to CORS, plus regression tests). Deploy production succeeded 10:09 AM PT. Live preflight now returns access-control-allow-methods: GET,POST,PUT.
#231	M1 release-day polars ComputeError (null/int rows then float 334.576909) at 3:35 AM PT Oct 16, failed again on both retries. Cause of the 2 invdash-job-dlq messages (invdash-dlq-not-empty ALARM) and invdash-lambda-errors 3:51–3:54 AM PT.	Fixed and verified (Oct 17). Merged #236 (89c20de, 9:27 PM PT Oct 16); deployed with Deploy run 39003455688 (success 9:48 PM PT). Live check: M1 run M1-20261017T162301Z-e498c76d at 9:23 AM PT Oct 17 finished success (13 sources, 0 failed); no release-day errors since 3:38 AM PT Oct 16. invdash-dlq-not-empty is still ALARM because the 2 Oct 16 messages remain in invdash-job-dlq; purge needs Ram's OK.
#232	DS-41 news-sentiment: Finnhub company-news 403 for fund ARKVX every hourly run (25/25 Oct 15–16); Finnhub key appeared in logged error URLs (already in CloudWatch Logs; rotation is Ram's call). Fix skips is_fund symbols, treats 403 for non-equities as a skip, and redacts query credentials.	Fixed and verified (Oct 17). Merged #237 (cc92648, 9:27 PM PT Oct 16); deployed with Deploy run 39003455688 (success 9:48 PM PT). Live check: H2 run at 9:22 AM PT Oct 17 finished success with 0 failed sources, and no 403 or ARKVX log lines since the deploy. Finnhub key rotation is Ram's call.
#233	invdash-failed-runs double-counts one hourly DS-41 failure when datapoints land at :19 then :18 (ALARM about 1 min at 12:19 and 9:19 PM PT Oct 15; 12:19, 3:19, 7:19 and 10:19 AM PT Oct 16). Fix counts distinct failing sources (FailedSources by source_id, Terraform via PR only).	Fixed (Oct 17). #238 deploy failed 9:53 PM PT Oct 16 (SEARCH is not supported on Metric Alarms); replaced by #241, merged 9:21 AM PT Oct 17 (0a271aa) and deployed with Deploy run 39049652305 (success 9:37 AM PT). invdash-failed-runs is now a plain metric alarm on FailedSources and is OK.
#234	DS-05 price-reconcile: Yahoo chart 404 for SYFY at 4:15 PM PT Oct 15 (with DS-41, the 4:16 PM PT alarm). Fix marks unresolved symbols as a warning. SYFY may be a typo for SYFI; Ram to decide.	Fixed (Oct 16). Merged #239 (a4d58c5, 9:28 PM PT); deployed with Deploy run 39003455688 (success 9:48 PM PT). SYFY vs SYFI: Ram to decide.
#235	D5 gov-contracts: NASA feed 429 at 2:45 PM PT Oct 15 (with DS-41, the 2:46 PM PT alarm). #217 fix held: defense.gov is a warning; no SAM.gov 400s. Fix retries with Retry-After backoff, then warns.	Fixed (Oct 16). Merged #240 (417fa97, 9:29 PM PT); deployed with Deploy run 39003455688 (success 9:48 PM PT).

• Nothing came in from real people since the 11:09 AM PT check.
• The only new mail was AWS CloudWatch alarm emails (US West, N. California).
• No emails were changed during the review itself.
• Update Oct 14: Causes of several alarms in this report are now tracked in #182–#184 (see Tracked issues). The recurring serving-read-failed errors (R1) map to the prefs-api chart route permission gap in #183, and the invdash-lambda-errors flaps (A1) are driven by the regulatory-feeds failure in #182. Later Oct 14: #182–#184 are closed; fix PRs #185–#189 are merged. Oct 14 ~3:16 PM PT: regulatory-feeds succeeding post-#186; DLQ purged (was empty); invdash-dlq-not-empty OK. invdash-failed-runs still ALARM from gov-contracts SAM/DoD provider errors (not #182). Evening Oct 14: gov-contracts cause tracked in #216, fixed by #217 (merged, deployed); invdash-failed-runs OK since 5:46 PM PT, pending the Oct 15 2:45 PM PT run to confirm. Oct 15 evening: that run and later hours still produced FailedRuns ≥ 2 flaps (see #216 row); currently OK as of ~5:16 PM PT. Source identity not known from SNS email alone.

Alarm	What happened (PT)	Status at review
invdash-prefs-api-serving-read-failed	Went into ALARM at 11:42 AM. A later check showed it went OK at 11:58 AM and back into ALARM at 12:00 PM.	Active (ALARM)
invdash-site-api-5xx	Tripped at 11:47 AM and recovered at 11:51 AM.	Recovered (OK)

Both alarms have been flapping between ALARM and OK all morning, starting around 7:21 AM PT. The prefs API serving-read failure is the one that still needs attention.

Update Oct 14: #183 found that SERVING_READ_FAILED on GET /chart/{ticker} comes from a missing serving/chart_data/{TICKER}.json returning AccessDenied (no ListBucket on that prefix). Fix in PR #185 (merged, #183 closed). See R1.

• AWS alarm emails were archived to All Mail (Gmail's default archive). Their read/unread state wasn't changed. That covered 11 threads, including the invdash-prefs-api-serving-read-failed and invdash-site-api-5xx alarms and their OK notices, plus older invdash-lambda-errors and invdash-dashboard-payload-size notifications.
• The only email left in the inbox is the Ripple interview message.
• No personal mail was touched.

In the review window (11:09 AM to about 12:03 PM PT):

Category	Emails	Notes
AWS alerts (CloudWatch)	5	3 ALARM and 2 OK: invdash-prefs-api-serving-read-failed and invdash-site-api-5xx
AWS alerts (Security Hub, Inspector, CloudTrail)	0	None this window
Personal	0	None
Other (people, companies, recruiters, bills)	0	None

AWS threads archived after the review (11 threads, all from [email protected]):

Category	Threads	Messages in those threads*	Issue
Inspector2 findings	2	about 110	S1 (vulnerable invdash/jobs images)
Security Hub findings (imported Inspector copies)	2	about 200	S1 and S5 (duplicate alert email)
CloudWatch invdash-prefs-api-serving-read-failed (ALARM and OK)	2	14	R1 (new, recurring)
CloudWatch invdash-site-api-5xx (ALARM and OK)	2	19	A2 (recurring)
CloudWatch invdash-lambda-errors (ALARM 7:20 AM, OK 8:55 AM PT)	2	2	A1 (recurring)
CloudWatch invdash-dashboard-payload-size (new alarm, INSUFFICIENT_DATA to OK)	1	1	Informational, same missing-data caveat as A3

*These are total message counts for each thread as Gmail reports them. Busy threads cap at 100, and some messages were archived earlier this morning.

Update Oct 14 (missing OK emails): Per #182, invdash-alert-notifier sends at most 1 email per alarm per 60 min and its dedup key ignored ALARM vs OK, so recoveries within the hour were suppressed (alert_suppressed). Some ALARM emails above may therefore have had no matching OK email. PR #188 (open) adds the state to the dedup key; see S5.

Inbox after archiving: 1 email in the "Other" category (the Ripple interview message), 0 AWS and 0 personal.

The security findings in today's threads are the same 16 Inspector CVEs as S1 below. No new finding types showed up.

Each security issue, configuration issue, and recurring error has its remediation steps below. R1 is new today. Everything after it comes from the AWS-only report prepared at about 6:20 AM PT today, and its steps still apply. The statuses in those sections are as of 6:20 AM. Since then, invdash-site-api-5xx (A2) has flapped several times this morning and was last OK at 11:51 AM PT, and invdash-lambda-errors (A1) alarmed again at 7:20 AM and cleared at 8:55 AM PT. Both are now recurring errors, not one-offs.

Evidence from email:

• Custom metric invdash/prefs-api SERVING_READ_FAILED, Sum ≥ 1 in one 5-minute period.
• 7 ALARM emails today. Error counts were 1.0 at 6:17 AM, 6.0 at 6:46 AM, 2.0 at 7:48 AM, 1.0 at 8:14 AM, 2.0 at 9:12 AM, 3.0 at 11:37 AM and 1.0 at 11:55 AM PT (datapoint times).
• Each ALARM cleared to OK within a few minutes, then came back.

What needs to be fixed:

• The prefs API periodically fails to read the dashboard serving document from the data lake (S3, KMS-encrypted).
• Short bursts that keep coming back point to an intermittent cause, not a hard outage. Possible causes are a missing or partially written serving object during a job's publish, an S3 or KMS permission gap from the narrowed inline policies (see S6), or timeouts on large documents.
• (Unverified): the actual cause. It isn't in the email.
• Update Oct 14: For the chart route, the cause is now known (#183, extends #178). The prefs-api role can GetObject on serving/chart_data/* but had no s3:ListBucket for that prefix, so a missing serving/chart_data/{TICKER}.json returned AccessDenied and surfaced as 502 SERVING_READ_FAILED (six such requests for one user on Oct 13 around 7:26 PM PT). #178 only covered serving/stock/*. Whether every Oct 10 burst came from this route is not verified.

Update Oct 14, fix in PR #185 (merged, #183 closed):

• Terraform: s3:ListBucket with prefix serving/chart_data/* on the prefs-api role (same pattern as #178), so a missing chart reads as NoSuchKey.
• Code: NoSuchKey / 404 / AccessDenied-on-missing for chart data maps to a clean 404 CHART_NOT_READY instead of SERVING_READ_FAILED. KMS denials and throttling still surface as SERVING_READ_FAILED.
• Logging: prefs_api_error on GET /chart/{ticker} now includes the ticker.
• Deploy via the normal pipeline after merge; remediation step 3 below is covered for this prefix by #185.

Remediation steps:

1. Find the error class in the prefs API Lambda logs for the burst windows (for example 6:40–6:50 AM PT, which is 13:40–13:50 UTC):

   fields @timestamp, @message
   | filter @message like /SERVING_READ_FAILED|AccessDenied|NoSuchKey|KMS|timed out/
   | sort @timestamp desc
   

2. Line the bursts up with job schedules and serving-document writes. If reads fail while a job rewrites the object, publish atomically: write to a temp key, then copy or rename it, or version the key and switch a pointer. Have the API fall back to the last good document.
3. If the logs show AccessDenied or a KMS error, add the exact S3 prefix or kms:Decrypt (with kms:ViaService) to the invdash-prefs-api role in Terraform.
4. Add retries with backoff on the S3 read, and cache the last good document in memory so a single failed read doesn't surface as an error.
5. Reduce flapping: use 2 of 2 periods, or alarm on an error rate rather than a single count, and keep a separate alarm for sustained failure.
6. Verify: No serving-read-failed ALARM emails for 24 hours across job runs, and the Logs Insights query returns 0 rows.

None found. The AWS email has no billing, invoice, AWS Budgets, Cost Anomaly Detection or Free Tier alerts.

• What needs to be fixed: Ram has no cost alerting reaching this mailbox. It's either not set up or goes somewhere else. (Unverified: whether a Budget exists can't be told from email.)
• Remediation steps:
  1. Create a monthly cost budget with email alerts at 50%, 80% and 100% of actual spend, and at 100% of forecast. In Terraform:

     resource "aws_budgets_budget" "monthly" {
       name         = "invdash-monthly"
       budget_type  = "COST"
       limit_amount = "50"        # pick your number
       limit_unit   = "USD"
       time_unit    = "MONTHLY"
       notification {
         comparison_operator        = "GREATER_THAN"
         threshold                  = 80
         threshold_type             = "PERCENTAGE"
         notification_type          = "ACTUAL"
         subscriber_email_addresses = ["[email protected]"]
       }
       notification {
         comparison_operator        = "GREATER_THAN"
         threshold                  = 100
         threshold_type             = "PERCENTAGE"
         notification_type          = "FORECASTED"
         subscriber_email_addresses = ["[email protected]"]
       }
     }

  2. Turn on Cost Anomaly Detection: an AWS-services monitor plus a daily email subscription.
  3. Verify: AWS Budgets sends no confirmation email, so check the budget appears in the Budgets console. An anomaly-detection subscription produces alert emails only when it finds an anomaly.

---

• Source emails: 135 "Inspector2 Finding" emails, plus 135 identical "Security Hub Findings - Imported" copies (270 total). Fri Oct 9, 2:08 PM PT → Sat Oct 10, 6:02 AM PT.
• Resource: ECR repo invdash/jobs (arm64, Amazon Linux 2023). 9 image pushes were scanned: Oct 9 at 2:07 PM, 8:44 PM, 10:41 PM, 10:51 PM and 11:44 PM; Oct 10 at 12:46 AM, 1:07 AM, 1:13 AM and 6:01 AM PT (latest, tag 4f3c257f9b32eca99c80c868b47307bd38a61af2-1).
• Pattern:
  • The first image (2:07 PM, Oct 9) had 6 findings: rpm ×2, curl, pcre2 and chi ×2.
  • From the 8:44 PM image on, every image has 16. The extra 10 are go/stdlib CVEs in the same aws-lambda-rie binary that already carried the chi findings.
  • That suggests Inspector's vulnerability data picked up the new Go CVEs, or the emulator binary changed. (Unverified: which of the two.)
  • All findings are ACTIVE; none has closed.

Severity	Finding	Where (from email)	Installed → fixed	Score	Notes from finding text
CRITICAL	CVE-2026-56857	go/stdlib in /usr/local/bin/aws-lambda-rie	1.26.7 → 1.27.2	9.8	os.Root.Mkdir junction escape. Affects Windows only, so it can't be exploited on this Linux image, but it still counts against compliance.
CRITICAL	CVE-2026-78663	go/stdlib, same binary	1.26.7 → 1.27.2	9.1	HTTP/2 server double-refunds flow control
HIGH	CVE-2026-56866, -78659, -78660, -78667, -78669, -94439, -94440, -97031	go/stdlib, same binary	1.26.7 → 1.27.2	7.5	net/http, HTTP/2, multipart and ECH denial-of-service or request-handling issues
HIGH	GHSA-rjr7-jggh-pgcp	github.com/go-chi/chi/v5, same binary	v5.2.4 → 5.3.0	n/a	realip middleware trusts X-Forwarded-For
HIGH	GHSA-9g5q-2w5x-hmxf	github.com/go-chi/chi/v5, same binary	v5.2.4 → 5.3.0	n/a	RemoteAddr can be spoofed via X-Forwarded-For
HIGH	CVE-2026-80230 (exploit available: YES)	curl-minimal, libcurl-minimal (OS)	8.21.0-5.amzn2023.0.1 → 8.21.0-5.amzn2023.0.2	7.4	Public-key pinning isn't enforced when VERIFYPEER/VERIFYHOST are off
HIGH	CVE-2026-89161	pcre2, pcre2-syntax (OS)	10.40-1.amzn2023.0.3 → 10.40-1.amzn2023.0.4	7.4	JIT match incorrect free
HIGH	CVE-2026-78367	rpm, rpm-libs (OS)	4.16.1.3-29.amzn2023.0.7 → 4.16.1.3-29.amzn2023.0.8	7.8	rpmbuild tarball macro injection
HIGH	CVE-2026-84233	rpm, rpm-libs (OS)	→ 4.16.1.3-29.amzn2023.0.8	7.0	rpmuncompress -x macro injection

What needs to be fixed:

• (a) The production image ships the Lambda Runtime Interface Emulator (/usr/local/bin/aws-lambda-rie). It's built with Go 1.26.7 and chi v5.2.4, which accounts for 12 of the 16 findings, including both CRITICALs.
  • Per AWS, the emulator "is included in all AWS-provided base images" and exists for local testing. When the function runs in Lambda, the real Runtime API is used instead.
• (b) The AL2023 OS packages in the image are one patch release behind.

Remediation steps:

1. Find where the emulator comes from. Check whether the Dockerfile starts FROM public.ecr.aws/lambda/provided:al2023 (or another AWS Lambda base image), or whether it downloads aws-lambda-rie itself.
   • If it's a downloaded copy, delete that curl/ADD step from the production stage.
2. Remove the emulator from the image you deploy (preferred: smallest attack surface).
   • Do this in the final stage of a multi-stage build:

     FROM public.ecr.aws/lambda/provided:al2023 AS runtime
     # ...copy bootstrap / binaries...
     # The emulator is for local testing only; not needed when running in Lambda
     RUN rm -f /usr/local/bin/aws-lambda-rie

   • For local tests, mount the emulator at run time instead: docker run -v ~/.aws-lambda-rie:/aws-lambda --entrypoint /aws-lambda/aws-lambda-rie .... This follows the AWS Lambda Go container-image guide.
   • (Unverified): check that the base image's /lambda-entrypoint.sh only calls the emulator when AWS_LAMBDA_RUNTIME_API is unset. Then run one test invoke in Lambda after the change.
3. Or, if you need to keep the emulator: pull a newer base image digest, or the latest emulator release, built with Go ≥ 1.27.2 and chi ≥ 5.3.0.
   • (Unverified): whether AWS has already published an emulator or base image with those versions. Check the emulator's GitHub releases page and the base image's tag history before relying on this.
   • Pin the base image by digest and let Dependabot/Renovate update it:

     FROM public.ecr.aws/lambda/provided:al2023@sha256:<new-digest>

4. Patch the OS packages in the same stage:

   RUN dnf -y upgrade --releasever=latest \
         rpm rpm-libs curl-minimal libcurl-minimal pcre2 pcre2-syntax \
    && dnf clean all

   • The minimum fixed versions are rpm/rpm-libs 4.16.1.3-29.amzn2023.0.8, curl-minimal/libcurl-minimal 8.21.0-5.amzn2023.0.2 and pcre2/pcre2-syntax 10.40-1.amzn2023.0.4.
   • Add a build-time check, for example RUN rpm -q curl-minimal pcre2 rpm, and fail the build if any is older.
5. If your own Go code also uses chi (not shown by these findings, which point only at the emulator binary), bump it and rebuild with Go ≥ 1.27.2:

   go get github.com/go-chi/chi/[email protected] && go mod tidy
   # go.mod
   toolchain go1.27.2
   

   • Don't use middleware.RealIP unless the service is behind a proxy that overwrites X-Forwarded-For.
6. Gate future builds in CI so a vulnerable image never deploys. Add an image scan step in GitHub Actions, for example Inspector's aws-actions/vulnerability-scan-github-action-for-amazon-inspector, or Trivy/Grype as a stand-in, that fails on CRITICAL/HIGH findings that have a fix.
7. Verify the fix:
   • ECR enhanced scanning rescans automatically when you push. Push the rebuilt image and check that the "Inspector2 Finding" emails for the new tag stop.
   • In the Inspector console (Findings → by container image), filter on invdash/jobs and the new image digest. You should see 0 CRITICAL/HIGH.
   • Old findings on superseded image digests stay ACTIVE until those images are deleted. Expire untagged and old images with an ECR lifecycle policy so the backlog closes.
   • (Unverified): exactly when Inspector closes findings for deleted images.

• Source emails: 3, from Fri Oct 9, 3:31 AM PT to Sat Oct 10, 3:15 AM PT. First observed Mon Oct 6, 3:21 AM PT. Workflow status: NEW.
• Resource: AWS::::Account:308639168050, us-west-1.
• Reason (from email): "The AWS Config recorder is using a custom IAM role instead of the AWS Config service-linked role" (CONFIG_RECORDER_CUSTOM_ROLE). Control parameter: includeConfigServiceLinkedRoleCheck = true.

What needs to be fixed: Point the Config configuration recorder in us-west-1 at the service-linked role AWSServiceRoleForConfig. Per AWS documentation, custom roles "require manual policy management and might not automatically receive updates when AWS Config adds support for new resource types".

Remediation steps:

1. Create the service-linked role if it doesn't exist (this is a no-op if it does):

   aws iam create-service-linked-role --aws-service-name config.amazonaws.com
   

2. Update the recorder. In Terraform, which is where the custom role most likely came from:

   resource "aws_iam_service_linked_role" "config" {
     aws_service_name = "config.amazonaws.com"
   }
   resource "aws_config_configuration_recorder" "this" {
     name     = "default"
     role_arn = aws_iam_service_linked_role.config.arn
     recording_group {
       all_supported                 = true
       include_global_resource_types = true   # in one "home" region only
     }
   }

   Or with the CLI:

   aws configservice put-configuration-recorder --configuration-recorder \
     name=default,roleARN=arn:aws:iam::308639168050:role/aws-service-role/config.amazonaws.com/AWSServiceRoleForConfig \
     --recording-group allSupported=true,includeGlobalResourceTypes=true
   

3. Confirm the delivery channel still works (the S3 bucket policy must allow config.amazonaws.com), then delete the old custom Config role from Terraform.
4. If you really do need the custom role, for example to cut costs, the alternative is to set the control parameter includeConfigServiceLinkedRoleCheck=false or suppress the finding with a written justification.
5. Verify: Config.1 is a periodic control (per Security Hub docs). After the next evaluation, which can take up to about 24 hours, its status should be PASSED and the "Config.1" emails should stop.
   • Check with aws securityhub get-findings --filters '{"ComplianceSecurityControlId":[{"Value":"Config.1","Comparison":"EQUALS"}]}' (looking for Compliance.Status = PASSED), or in Security Hub → Controls → Config.1.
   • Per AWS docs, if the newer Security Hub (not CSPM) is enabled alongside CSPM, Config.1 always reports PASSED because it uses a service-linked recorder.

• Source emails: 2, from Fri Oct 9, 10:39 AM PT to Sat Oct 10, 4:38 AM PT. First observed Tue Oct 6, 2:37 PM PT. Workflow status: NEW.
• Resource: sg-0c1a882bceb196e9f (group name default) in VPC vpc-062cf5b40615b0450, us-west-1.
• Current rules (from email):
  • Ingress: all protocols from the group itself.
  • Egress: all protocols to 0.0.0.0/0.
  • These are the AWS default rules.

What needs to be fixed: Remove every inbound and outbound rule from the VPC's default security group. Make sure nothing (Lambda in the VPC, endpoints, RDS, etc.) relies on it; anything that does should get its own least-privilege group.

Remediation steps:

1. List the network interfaces that use the default group: aws ec2 describe-network-interfaces --filters Name=group-id,Values=sg-0c1a882bceb196e9f (read-only).
   • Move each one to a dedicated security group first, following the remediation text in the Security Hub email.
2. Lock down the default group in Terraform. Declaring aws_default_security_group with no ingress or egress blocks removes all rules:

   resource "aws_default_security_group" "default" {
     vpc_id = "vpc-062cf5b40615b0450"   # or aws_vpc.main.id
     # no ingress/egress blocks => all rules removed
     tags = { Name = "default-locked-down" }
   }

   Or with the CLI:

   aws ec2 revoke-security-group-ingress --group-id sg-0c1a882bceb196e9f \
     --ip-permissions 'IpProtocol=-1,UserIdGroupPairs=[{GroupId=sg-0c1a882bceb196e9f}]'
   aws ec2 revoke-security-group-egress --group-id sg-0c1a882bceb196e9f \
     --ip-permissions 'IpProtocol=-1,IpRanges=[{CidrIp=0.0.0.0/0}]'
   

3. Do the same for the default security group of every VPC in every region you use.
4. Verify: In Security Hub → Controls → EC2.2, the resource should show PASSED after re-evaluation, and the "EC2.2" emails should stop.
   • (Unverified: how quickly EC2.2 re-evaluates. Expect hours, not minutes.)
   • Run a Lambda test invoke to confirm nothing lost network access.

• Source email: 1 "AWS Console Sign In via CloudTrail" (aws.signin), already archived, in one of the 12 threads.
• Details: Thu Oct 8, 10:02 PM PT. ConsoleLogin: Success, user arn:aws:iam::308639168050:root, MFA used: Yes (U2F security key Securitykey-W47ALYXECZFJVNVPIGI5GRUD64). IP 172.225.89.50 (Apple iCloud Private Relay range); AWS Console iOS app 3.23.0 (Safari user agent).

What needs to be fixed: Nothing suggests compromise, but root was used for a session from a phone app. Root should be used only for the few tasks that require it. IAM Identity Center is already set up for this account (invitation and email verification on Oct 6), so use that for day-to-day access.

Remediation steps:

1. Confirm Ram made this sign-in from his iPhone around 10:00 PM PT on Oct 8. If not, treat it as critical: change the root password, review CloudTrail for root activity after it, and rotate everything.
2. Use the IAM Identity Center user (with MFA) in the AWS Console app instead of root. Sign out of root on the phone.
3. Make sure root has no access keys: aws iam get-account-summary should show AccountAccessKeysPresent: 0. Register a second hardware MFA device on root as a backup.
4. Keep the existing EventBridge rule that emails on root sign-in. It worked. Optionally add a rule for root API calls (userIdentity.type = Root) as well.
5. Verify: Over the next weeks, there should be no root "Console Sign In" emails except ones Ram knowingly triggers.

• What needs to be fixed: The raw EventBridge JSON is being emailed to a personal Gmail address. It includes account ID, role ARNs, source IPs and session tokens such as the secureToken login URL parameter in the sign-in event. It also arrives in high volume: 309 emails in about 2 days, mostly duplicates.
• Remediation steps:
  1. Stop routing "Security Hub Findings - Imported" to email for findings that come from Inspector (ProductName = Inspector). They duplicate the Inspector2 emails one for one.
     • Change the EventBridge rule pattern so it only forwards Security Hub's own control findings at CRITICAL/HIGH with Workflow.Status = NEW.
  2. Use an EventBridge input transformer to send a short message (title, severity, resource, link) instead of the full JSON.
  3. Optionally send a daily digest instead of per-finding emails.
  4. Verify: For the next image push, there should be one concise email per new finding, not two raw-JSON emails.

Update Oct 14, CloudWatch alarm email noise (#182):

• The opposite problem also exists: invdash-alert-notifier (only subscriber of invdash-ops-alarms) dedups on alarm:<name> with 1 message per 60 min (EMAIL_MAX_PER_WINDOW=1, EMAIL_WINDOW_MINUTES=60), ignoring ALARM vs OK. OK emails arriving within the hour were suppressed, so operations got 7 ALARM emails overnight and no recoveries.
• PR #188 (merged, part of closed #182): the key becomes alarm:<name>:<STATE> (ALARM / OK / INSUFFICIENT_DATA); repeats of the same state stay capped. Side effect: the key is shared by both channels, so SMS will also send one recovery text per incident. Email-only recoveries would need a follow-up keyed on channel.
• Alarm causes behind the current noise (#182):
  • invdash-failed-runs: the release-day Census source asked for BOPG with seasonally_adj=yes, which doesn't exist, so Census returned an empty body (JSONDecodeError). PR #187 (merged) switches to BOPGS seasonally adjusted and treats a 204/empty body as no data (census_no_data, 0 rows, not a failed run).
  • invdash-dlq-not-empty: regulatory-feeds polars schema-inference ComputeError on insider-name columns; fixed in PR #186 (merged, see A1).
  • After #186 deploys, invdash-job-dlq must be purged (needs Ram's OK) and both alarms confirmed back to OK.

• Source emails: 41 "AWS API Call via CloudTrail" emails in two bursts:
  • Fri Oct 9, 10:51–10:52 PM PT: 7 calls, run gha-deploy-38028763772-1, IP 172.214.47.40.
  • Sat Oct 10, 6:02 AM PT: 34 calls, run gha-deploy-38053854396-1, IP 20.168.124.211.
• Principal: role invdash-terraform-deploy, assumed via GitHub OIDC (token.actions.githubusercontent.com). User agent: Terraform 1.16.5 / aws provider 6.67.0.
• Calls:
  • 25 × PutRolePolicy (inline policies). Roles: invdash-prefs-api, invdash-jobs-scheduler, invdash-job-status-feed, and the 18 invdash-job-* roles.
  • 16 × AttachRolePolicy, attaching invdash-prefs-collector-read.
  • None of: CreateAccessKey, CreateUser, AttachUserPolicy, admin or *:* grants, or errors.
• Inline policies seen: tightly scoped. Logs go to each function's own log group, S3 access is limited to specific invdash-lake-308639168050 prefixes, KMS use is limited to one key with kms:ViaService, and events:PutEvents is limited to events:source = invdash.jobs. Some statements use Resource: "*" (most likely X-Ray, which needs it; unverified per statement).
• iam:PassRole on invdash-job-release-day (policy create-one-off-schedules) is already scoped correctly:

  {"Action":"iam:PassRole","Effect":"Allow",
   "Resource":"arn:aws:iam::308639168050:role/invdash-jobs-scheduler",
   "Condition":{"StringEquals":{"iam:PassedToService":"scheduler.amazonaws.com"}}}

  Its scheduler permissions are limited to arn:aws:scheduler:us-west-1:308639168050:schedule/default/invdash-m1-*.

What needs to be fixed: No changes are needed for PassRole. The remaining hardening is about the deploy role itself and the Resource: "*" statements.

Remediation steps:

1. Keep the PassRole statement exactly as it is: a specific role ARN plus iam:PassedToService. Add a policy test, for example a checkov or tfsec rule or an IAM Access Analyzer ValidatePolicy check in CI, that fails if any policy grants iam:PassRole on * or without iam:PassedToService.
2. Review each Resource: "*" statement. Only actions that don't support resource-level permissions (for example xray:PutTraceSegments and xray:PutTelemetryRecords) should use *.
3. Harden invdash-terraform-deploy:
   • Make sure its OIDC trust policy limits token.actions.githubusercontent.com:sub to your repo and the main branch or environment, for example repo:<owner>/<repo>:environment:prod.
   • Add a permissions boundary that every role it creates must carry, so a compromised pipeline can't create an admin role.
4. Expect these emails on every deploy. To cut noise, filter the EventBridge rule to sensitive events only: CreateAccessKey, CreateUser, AttachUserPolicy, PutUserPolicy, UpdateAssumeRolePolicy, CreateLoginProfile, any call where userIdentity.type = Root, and any call not made by invdash-terraform-deploy.
5. Verify:
   • Run aws accessanalyzer validate-policy --policy-type IDENTITY_POLICY --policy-document file://policy.json on the rendered policies in CI.
   • After the next deploy, only filtered IAM events should arrive by email.

Update Oct 14 (#183): The narrowed invdash-prefs-api inline policy was missing s3:ListBucket for serving/chart_data/*, which turned missing chart objects into AccessDenied (R1). PR #185 adds ListBucket scoped to that prefix only, keeping the policy tightly scoped. Expect the corresponding PutRolePolicy CloudTrail email when it deploys.

---

Alarm (us-west-1)	Metric / threshold (from email)	Transition history (PT)	Status
invdash-lambda-errors: "Any job Lambda errored in two consecutive 15-minute windows"	AWS/Lambda Errors, Sum ≥ 1 for 2 of 2 × 900 s (no dimensions, so it covers every function)	Oct 8 10:00 PM ALARM→OK (no earlier ALARM email in mailbox) · Oct 9 2:01 PM OK→ALARM (3.0 @1:31 PM, 2.0 @1:46 PM) · 2:03 PM →OK · 4:31 PM OK→ALARM (4.0 @4:01 PM, 5.0 @4:16 PM) · 4:34 PM →OK	OK (recurring)
invdash-site-api-5xx: "Site API returned 5xx errors in two consecutive 5-minute windows"	AWS/ApiGateway 5xx, Sum ≥ 1 for 2 of 2 × 300 s, ApiId rsr4pm3622, stage $default	Oct 9 11:58 PM OK→ALARM (1.0 @11:48 PM, 2.0 @11:53 PM) · 11:59 PM →OK (0.0 @11:54 PM)	OK
invdash-prefs-api-internal, -prefs-store-unavailable, -serving-read-failed, -serving-document-invalid, -ticker-added-publish-failed, -sts-unavailable	invdash/prefs-api custom metrics, Sum ≥ 1 in 1 × 300 s, missing data treated as NonBreaching	Oct 10 1:08–1:09 AM INSUFFICIENT_DATA→OK	OK. These are newly created alarms.

Source emails: 13 CloudWatch emails, from Oct 8, 10:00 PM PT to Oct 10, 1:09 AM PT. 8 were in the inbox (site-api-5xx ×2, prefs-api ×6). The 5 lambda-errors emails had already been archived.

What needs to be fixed:

• At least one scheduled job Lambda fails intermittently: 3 bursts in about 18 hours, with up to 5 errors per 15 minutes.
• The alarm has no FunctionName dimension, so the email can't say which function failed.
• Each ALARM cleared within about 3 minutes, which fits jobs that fail on a schedule rather than a sustained outage.
• (Unverified): the actual failing function and the error. Neither appears in any email.
• Update Oct 14: Per #182, invdash-regulatory-feeds fails hourly (:25) with polars ComputeError: could not append value: "Liu Teyin M" of type: str (schema inferred from the first 100 rows, mostly None names), and this drives every overnight invdash-lambda-errors flap; exhausted retries fill invdash-job-dlq. Not verified that it explains the Oct 9 bursts. PR #186 (merged, #182 closed) builds frames with infer_schema_length=None and casts name/role/code columns to Utf8, with a regression test. Oct 14 ~3:16 PM PT: confirmed hourly job_finished success since 9:25 AM PT after deploy; invdash-job-dlq purged (was already empty).

Remediation steps:

1. Find the failing function(s) with a CloudWatch Logs Insights query across /aws/lambda/invdash-*, for Oct 9 1:15–2:05 PM and 3:45–4:35 PM PT (20:15–21:05 and 22:45–23:35 UTC):

   fields @timestamp, @log, @message
   | filter @message like /(?i)(error|exception|panic|Task timed out|AccessDenied)/
   | stats count(*) by @log
   | sort count(*) desc
   

   Or use aws cloudwatch get-metric-data on AWS/Lambda Errors grouped by FunctionName for the same window.
2. Look specifically for AccessDenied or KMS/S3 permission errors. The Oct 9–10 deploys rewrote the inline policies of 18 job roles and narrowed S3 and KMS access, so a job touching a prefix outside its policy would fail this way.
   • Also check for timeouts (Task timed out) and upstream data or HTTP errors.
3. Fix the root cause:
   • Missing permission: add the specific prefix or action in Terraform.
   • Timeouts: raise timeout or memory_size, or batch smaller.
   • Flaky upstream: add retries with backoff, and send failed async invocations to an on-failure destination or DLQ (aws_lambda_function_event_invoke_config with destination_config.on_failure pointing to SQS).
4. Make the alarm actionable: split it per function, or add a metric-math or Logs-based alarm that names the function. Keep the aggregate alarm as a backstop.

   resource "aws_cloudwatch_metric_alarm" "job_errors" {
     for_each            = toset(var.job_function_names)
     alarm_name          = "invdash-${each.key}-errors"
     namespace           = "AWS/Lambda"
     metric_name         = "Errors"
     dimensions          = { FunctionName = each.key }
     statistic           = "Sum"
     period              = 900
     evaluation_periods  = 2
     threshold           = 1
     comparison_operator = "GreaterThanOrEqualToThreshold"
     treat_missing_data  = "notBreaching"
     alarm_actions       = [aws_sns_topic.ops_alerts.arn]
     ok_actions          = [aws_sns_topic.ops_alerts.arn]
   }

5. Verify: No invdash-lambda-errors ALARM emails for at least 48 hours, covering every job schedule. The Logs Insights query above returns 0 rows for the new window.

What needs to be fixed:

• The Site API (HTTP API rsr4pm3622, stage $default) returned 3 × 5xx between 11:48 and 11:53 PM PT on Oct 9.
• That was about 4–9 minutes after an invdash/jobs image push (11:44 PM PT) and about 1 hour after a Terraform IAM deploy (10:51 PM PT). It recovered by 11:54 PM PT.
• This points to errors during rollout, such as a cold start on the new version or a permission or config change taking effect, with no automated health gate.
• (Unverified): the exact cause. The API's backend logs aren't in email, and it's not confirmed that the Site API runs from the invdash/jobs image.

Remediation steps:

1. Diagnose: turn on or check HTTP API access logs ($context.status, $context.integrationErrorMessage, $context.integration.error) and the backing Lambda's logs for 11:45–11:55 PM PT Oct 9 (06:45–06:55 UTC Oct 10).
   • Look for the same new prefs-api error classes (STS_UNAVAILABLE, PREFS_STORE_UNAVAILABLE, SERVING_READ_FAILED). Those alarms were created in the next deploy, so they were not yet watching at 11:48 PM.
2. Deploy via an alias with gradual traffic shifting and automatic rollback, using CodeDeploy for Lambda with a pre-traffic hook:

   resource "aws_lambda_alias" "live" {
     name             = "live"
     function_name    = aws_lambda_function.site_api.function_name
     function_version = aws_lambda_function.site_api.version
   }
   # Point the API Gateway integration at the alias ARN, not $LATEST.
   resource "aws_codedeploy_deployment_group" "site_api" {
     app_name               = aws_codedeploy_app.site_api.name
     deployment_group_name  = "site-api"
     service_role_arn       = aws_iam_role.codedeploy.arn
     deployment_config_name = "CodeDeployDefault.LambdaCanary10Percent5Minutes"
     deployment_style {
       deployment_type   = "BLUE_GREEN"
       deployment_option = "WITH_TRAFFIC_CONTROL"
     }
     auto_rollback_configuration {
       enabled = true
       events  = ["DEPLOYMENT_FAILURE", "DEPLOYMENT_STOP_ON_ALARM"]
     }
     alarm_configuration {
       enabled = true
       alarms  = ["invdash-site-api-5xx", "invdash-prefs-api-internal"]
     }
   }

3. Add a post-deploy smoke test in the GitHub Actions job. Call a /health endpoint that exercises STS, the prefs table, and the S3/KMS serving read. Fail the job and roll back the alias to the previous version if it doesn't return 200 within N retries.
4. Reduce cold-start errors: set the integration timeout and the Lambda timeout high enough. Optionally add provisioned concurrency on live during deploys.
5. Order deploys so IAM and policy changes are applied and given time to propagate before new code that depends on them goes live.
6. Verify:
   • On the next deploy, CodeDeploy shows the canary step passing.
   • No invdash-site-api-5xx ALARM email arrives.
   • Test the rollback once with a deliberately failing pre-traffic hook in a non-production stage.

What needs to be fixed:

• All six alarms went from INSUFFICIENT_DATA to OK when they were created during the 1:07 AM PT deploy. They aren't incidents.
• The catch: they treat missing data as not breaching, so if the code stops emitting these metrics, for example after a broken deploy, the alarms stay silently OK.

Remediation steps:

1. Keep treat_missing_data = "notBreaching" for these error-count metrics. That's correct for "count of errors" metrics.
2. Add a heartbeat or traffic alarm so silence is caught: for example, alarm when API Gateway Count for rsr4pm3622, or a custom REQUESTS metric, is < 1 over 30 minutes during expected traffic hours, with treat_missing_data = "breaching".
3. Verify: Trigger a test error in a non-production stage, or publish a test data point with aws cloudwatch put-metric-data --namespace invdash/prefs-api --metric-name INTERNAL --value 1. Confirm the ALARM and then OK emails arrive.

---

• Update Oct 14, news-sentiment (#184): Finnhub returned empty lists for every ticker around 6:16–7:27 PM PT Oct 13 (recovered around 11:16 PM), and morning runs had 2 of 12 tickers with 0 headlines (tickers not logged). Likely cause: from/to covered a single Eastern date. PR #189 (merged, #184 closed) widens the window to at least yesterday through today in America/New_York (older catch-up watermark still widens further), logs finnhub_fetch with ticker, rows, from, to, and leaves RSS / Massive / Alpha Vantage unchanged. No alarm in this report was tied to it.

• SNS subscription confirmations (5 emails, already archived): Tue Oct 6, 2:22 AM PT (×3) and Thu Oct 8, 8:46 AM PT (×2). Topics: invdash-ops-alerts (us-west-1), invdash-security-alerts (us-west-1), invdash-ops-alerts-us-east-1.

  • Alerts are arriving, so the us-west-1 topics are confirmed. (Unverified): whether the us-east-1 topic subscription was confirmed. No us-east-1 alarm emails were seen, but the IAM and sign-in events come from us-east-1 EventBridge.
  • Next step: In SNS, check that each subscription shows Confirmed, not PendingConfirmation.

• AWS IAM Identity Center: invitation and "Verify your email" for account 308639168050 (Tue Oct 6, 1:51–1:52 AM PT). Informational. This is the identity to use instead of root (see S4).

• AWS Health / Personal Health Dashboard, GuardDuty, Trusted Advisor: none found in email.

  • What needs to be fixed: GuardDuty findings aren't reaching email. Either GuardDuty is off or nothing is routed. (Unverified from email alone.)
  • Remediation steps:
    1. Enable GuardDuty in us-west-1 and us-east-1.
    2. Add an EventBridge rule on source = aws.guardduty with severity >= 7 → invdash-security-alerts.
    3. Add an AWS Health rule on source = aws.health → invdash-ops-alerts.
    4. Verify: aws guardduty create-sample-findings produces one email.