How alert notifications are configured in ram290476/investor-dashboard as deployed in production: the per-issue caps on email and SMS, and every other alert setting (recipients, SNS topics, CloudWatch alarms, metric filters, EventBridge rules, structured log events), with steps to add a new alert.

Status (Sun Oct 11 2026, 11:20 AM PT).

• PR #153 (email and SMS caps) and PR #157 (the plan-time count fix and the topic layout below) are merged as 78dbb17. Deploy production #200 applied them (10:43–10:46 PT).
• Email is capped. SMS is off because alert_sms_e164 is not set.
• The existing confirmed email subscriptions were kept, so nobody had to confirm anything again.
• PR #158 (one email per CVE across Inspector and Security Hub) is merged as 9a9dba0 and was deployed by Deploy production #201 (11:11–11:15 PT). VULN_WINDOW_MINUTES=1440 is set on the notifier. The scan of #201's image at 11:14 PT sent 16 emails for 16 CVEs and held back the 16 Security Hub copies (alert_suppressed). After #200 the same scan had sent 32.

Contents

1. Per-issue caps: email and SMS
2. What happens to repeats
3. All other alert configuration

CloudWatch alarms (us-west-1)            ─┐
EventBridge rule aws-health              ─┼─> invdash-ops-alarms ─────────┐
                                          │                               │
EventBridge security rules (8)           ─┼─> invdash-security-alarms ────┼─> invdash-alert-notifier (Lambda, us-west-1)
API key rotation check Lambda            ─┘                               │      │ caps per issue and channel
                                                                          │      │ (DynamoDB invdash-alert-cap)
CloudFront 5xx alarm (us-east-1) ──> invdash-ops-alerts-us-east-1 ────────┘      │
                                                                                 ├─> invdash-ops-alerts ──────> email (alert_emails)
                                                                                 ├─> invdash-security-alerts ─> email (security_alert_emails)
                                                                                 └─> invdash-sms ─────────────> SMS (only when alert_sms_e164 is set)

• Intake topics (invdash-ops-alarms, invdash-security-alarms, invdash-ops-alerts-us-east-1): alarms and rules publish here. Their only subscriber is the notifier Lambda.
• Delivery topics (invdash-ops-alerts, invdash-security-alerts, and optionally invdash-sms): the inboxes and handset subscribe here. Only the notifier publishes here. These are the original topics, and the email subscriptions created before #153 are still on them.
• There are no *-email topics. An earlier version of #153 proposed invdash-ops-email / invdash-security-email. #157 replaced that design before anything was applied.

Each alert issue sends at most 1 email per window and at most 1 SMS per window (default window: one Pacific clock hour). The two channels are counted separately, each with its own window length and maximum, so changing one does not affect the other.

Set these in infra/terraform/terraform.tfvars, which is git-ignored. In production the deploy workflow writes this file from the TERRAFORM_TFVARS secret. Commented examples are in infra/terraform/terraform.tfvars.example. The variables are declared in infra/terraform/variables.tf, passed through module "alerting" in infra/terraform/main.tf, and declared again in infra/terraform/modules/alerting/sms.tf.

Example terraform.tfvars:

alert_emails         = ["[email protected]"]
email_window_minutes = 60 # fixed Pacific clock buckets from local midnight; 60 = one clock hour
email_max_per_window = 1  # emails per issue per window
alert_sms_e164       = "+1XXXXXXXXXX" # optional
sms_window_minutes   = 60 # independent of email_window_minutes
sms_max_per_window   = 1  # texts per issue per SMS window
vuln_window_minutes  = 1440 # one mail per CVE (Inspector + Security Hub merged) per Pacific day

Changes reach production through the Deploy production workflow, which runs on main after CI passes. The Lambda reads the values at cold start. A missing, non-numeric or < 1 value falls back to 60 and 1 (_positive() in sms_notifier.py).

Windows are fixed clock buckets in America/Los_Angeles, aligned to local midnight, not a rolling hour from the first message (window_bounds() in sms_cap.py).

• With the default 60 minutes, 14:00–14:59 PT is one window and 15:00 starts the next. An alert at 14:50 PT and another at 15:05 PT are in different windows, so both can be sent.
• If the window length does not divide 1440, the last bucket of the day is shorter.
• The window id is the start time with its UTC offset (for example 2026-10-11T14:00-0700), so the repeated 1 AM hour on a fall-back day counts as two windows.

Routing (email_topic_arn() in sms_notifier.py). A message is routed by the name of the intake topic that delivered it. Names ending in -security-alarms are emailed through invdash-security-alerts, to security_alert_emails, or to alert_emails when that list is empty. Everything else goes through invdash-ops-alerts to alert_emails, including invdash-ops-alarms and the us-east-1 CloudFront topic.

Important: subscribe inboxes only to invdash-ops-alerts / invdash-security-alerts, and the handset only to invdash-sms. A subscription on an intake topic (invdash-ops-alarms, invdash-security-alarms, invdash-ops-alerts-us-east-1) bypasses the cap. The topic policies allow only the notifier role, and the account owner, to publish to the delivery topics. AWS Budgets emails come from AWS Budgets directly and are not capped.

issue_key(subject, message) in sms_cap.py returns a stable id for one problem. Alarm state is not part of the key, so ALARM and OK notifications for the same alarm share one budget. Checked in this order:

Keys are whitespace-normalized and clipped to 240 characters.

Why vulnerability findings are merged (PR #158, deployed in #201). When Deploy production #200 pushed a new jobs image at 10:45 PT on Oct 11 2026, the scan reported 16 CVEs. Each one came twice: as an Inspector2 Finding event and as its Security Hub Findings - Imported copy. Under the old keys (inspector:<title> and securityhub:<title>) that was 32 security emails in two minutes. With PR #158 each CVE in a repository is one issue:

• 16 emails for that scan, one per CVE.
• Within the same Pacific day, rescans and new image pushes with the same CVE send nothing more. Every new image gets a new Inspector finding, which is why the digest is not part of the key.
• The next day's first email for that CVE shows how many repeats were held back.
• Because the window starts at midnight, a CVE seen at 23:50 PT and again at 00:10 PT sends 2 emails.
• A different CVE, or the same CVE in a different repository or function, is a separate issue.

Tests: services/data-jobs/tests/test_sms_cap.py with fixtures tests/fixtures/notifier/inspector2_finding_cve.json and securityhub_imported_cve.json. These are the 10:46 PT events for CVE-2026-56857 with the account id redacted.

Within one window, after an issue has used its *_max_per_window messages on a channel:

• Repeats are suppressed on that channel: no email and/or no text. Email and SMS are decided independently, and email is attempted before SMS. With the defaults, a flapping alarm sends one email and one text per clock hour instead of one per ALARM/OK transition.
• They are still logged. Each suppressed repeat increments suppressed on the DynamoDB item and writes a JSON line to /aws/lambda/invdash-alert-notifier. The original alarm/event is also unchanged: CloudWatch alarm history, EventBridge, and the source Lambda logs (for example the prefs_api_error lines in section 3.6) still record every occurrence.
• The next allowed message carries the suppressed count. The first message for that issue in a later window includes the count from the previous window:
  • Email: the subject gets (suppressed N repeats) (or 1 repeat), kept within SNS's 100-character subject limit, and the body ends with Suppressed N repeats in the previous window.
  • SMS: the text (up to 140 characters of subject/summary) gets (suppressed N repeats), capped at 300 characters. The count only carries over from the window immediately before (the item expires after that), and only if the issue recurs.
• Fail open. If the cap table cannot be read or updated, that channel is sent anyway and alert_cap_failed_open is logged. A failed SNS publish does not use up the slot; the invocation raises so SNS retries it.

Notifier log events (one JSON object per line, from _log() in sms_cap.py):

{"event": "alert_suppressed", "issue_key": "alarm:invdash-site-api-5xx", "channel": "email", "suppressed": 3, "window": "2026-10-11T14:00-0700"}

Logs Insights query for suppressed repeats (log group /aws/lambda/invdash-alert-notifier):

fields @timestamp, issue_key, channel, suppressed, window
| filter event = "alert_suppressed"
| sort @timestamp desc

Everything below is read from main at 9a9dba0 (Oct 11 2026) and matches production after Deploy production #201. Resource names use the project prefix invdash (Terraform variable project, default "invdash") and the primary region us-west-1.

The current recipient is [email protected]. Its subscriptions on invdash-ops-alerts and invdash-security-alerts are Confirmed, checked after Deploy production #200 at 10:47 PT. Each address confirms two subscriptions, one per delivery topic. Nothing is subscribed by email in us-east-1: #153 removed the old direct email subscription on invdash-ops-alerts-us-east-1, and CloudFront mail now arrives from invdash-ops-alerts. An unconfirmed subscription receives nothing.

To change recipients: edit alert_emails / security_alert_emails in the tfvars (the TERRAFORM_TFVARS secret for production) and deploy. Each new address then confirms its subscription emails.

All topics are KMS-encrypted (the data key, or the us-east-1 alerts key) and deny publishes over plain HTTP. Intake topics allow cloudwatch.amazonaws.com and events.amazonaws.com to publish only from this account; invdash-ops-alerts-us-east-1 allows CloudWatch only. Delivery topics allow publishes only from the notifier role and the account owner.

Alarm names are invdash-<key>. Every us-west-1 alarm sends both ALARM and OK notifications (alarm_actions and ok_actions) to invdash-ops-alarms. ALARM and OK for the same alarm share one issue key, so a flapping alarm sends at most one email per window instead of one per transition.

infra/terraform/modules/observability/main.tf (local.base_alarms, local.canary_alarm_defs, resource aws_cloudwatch_metric_alarm.this). Custom metrics are in namespace InvestorDashboard (metrics_namespace), dimension service = invdash, emitted by services/data-jobs/src/app/observability.py and dashboard_build.

* Canary alarms exist only when health_url is set (infra/terraform/variables.tf, default ""). It is empty in production, so the canary and these three alarms are off (15 alarms exist in us-west-1 today).

infra/terraform/modules/user_prefs/main.tf (site API / preferences Lambda):

infra/terraform/modules/us_east_1/main.tf:

The only log metric filters are in infra/terraform/modules/user_prefs/main.tf: local.prefs_api_error_alarms drives both aws_cloudwatch_log_metric_filter.prefs_api_error and aws_cloudwatch_metric_alarm.prefs_api_error (one filter and one alarm per code).

• Log group: the site API function's group (aws_cloudwatch_log_group.api_fn).
• Filter name: invdash-prefs-api-<code in lower-kebab>, for example invdash-prefs-api-serving-read-failed.
• Pattern: a JSON filter, { ($.event = "<event>") && ($.code = "<CODE>") }. A quoted-term pattern such as "event": "x" is rejected by PutMetricFilter, and services/data-jobs/tests/test_prefs_api_metric_filters.py checks the filters against what _log_event prints.
• Metric: name <CODE>, namespace invdash/prefs-api, value 1 per matching line.

modules/observability/main.tf has no metric filters. Job metrics come from Powertools embedded metrics (EMF) in services/data-jobs/src/app/observability.py, not from log filters. It also defines three saved Logs Insights queries: invdash/failed-source-runs, invdash/slowest-sources, invdash/trace-one-run.

AWS service events on the default bus are free, so these replace metric-filter alarms for security events. Rule name: invdash-<key>. Each rule's target is the intake topic shown.

IAM and root/global sign-in events only reach the us-east-1 bus. modules/us_east_1/main.tf forwards them to the us-west-1 default bus (output global_event_patterns), where iam-changes and risky-sign-in alert.

securityhub-high and inspector-high both fire for the same CVE, because Security Hub imports Inspector's findings. Since PR #158 (deployed in #201), both copies share one vuln:<ID>:<resource> key with a one-day window. Each CVE now emails once per Pacific day per repository: 16 emails for #201's image scan at 11:14 PT, where #200's scan at 10:46 PT had sent 32.

Also on the security intake topic: the API key rotation check Lambda (modules/api_keys/main.tf, code services/data-jobs/src/functions/key_rotation_check/key_rotation_check.py) runs daily (check_schedule). It publishes to invdash-security-alarms (issue key key-rotation) when a key is overdue, due soon (remind_days), or never stored (Mondays). The notifier then emails it through invdash-security-alerts.

Budget alerts (modules/observability/main.tf, aws_budgets_budget.monthly): AWS Budgets emails alert_emails directly at 80% actual and 100% forecast of monthly_budget_usd (default 40). These do not go through SNS and are not capped.

Site API (services/data-jobs/src/functions/prefs_api/prefs_api.py): _log_event(event_name, route, method, exc, code) prints one JSON object per line with fixed keys only (no request body, token, or exception text):

{"event": "prefs_api_error", "route": "GET /dashboard", "method": "GET", "error_class": "ClientError", "code": "SERVING_READ_FAILED"}

• event: prefs_api_error (handler failures) or ticker_added_publish_failed (from _queue_backfill).
• route: normalized, for example GET /prefs, GET /chart/{ticker}, GET /stock/{ticker}, or GET /unknown.
• method: an HTTP verb or UNKNOWN.
• error_class: class of the underlying cause (exc.__cause__) or of the exception itself.
• code: STS_UNAVAILABLE, SERVING_READ_FAILED, PREFS_STORE_UNAVAILABLE (from _client_failure by AWS operation), SERVING_DOCUMENT_INVALID, INTERNAL, or TICKER_ADDED_PUBLISH_FAILED.

Jobs (services/data-jobs/src/app/observability.py, Powertools logger): source_run (per source: source_id, started_at, outcome, rows, duration_ms, error_type, error), rate_headroom (provider, used, limit) and freshness (stale, stale_p1, p1_due). The saved queries filter on these.

Key rotation check: key_rotation_status (one per key) and key_rotation_summary (emailed plus counts). Logs are kept 400 days as rotation evidence.

Log retention is log_retention_days (default 400) and log groups are encrypted with the audit KMS key.

A. New error code in the site API (log line → filter → alarm):

1. In prefs_api.py, call _log_event("<event>", route, method, exc, "<NEW_CODE>") where the failure happens. Keep the fixed keys and add no free text.
2. In infra/terraform/modules/user_prefs/main.tf, add an entry to local.prefs_api_error_alarms:

   NEW_CODE = {
     event       = "prefs_api_error"
     description = "Site API <what went wrong>"
   }

   This creates the filter invdash-prefs-api-new-code, the metric invdash/prefs-api NEW_CODE, and an alarm on the ops intake topic (invdash-ops-alarms).
3. Extend services/data-jobs/tests/test_prefs_api_metric_filters.py if needed, then run pytest.

B. New metric alarm on a job or AWS metric:

1. Emit the metric (for jobs, metrics.add_metric(name="X", unit=..., value=...) through app/observability.py so it gets the service dimension), or pick an existing AWS metric.
2. Add an entry to local.base_alarms in infra/terraform/modules/observability/main.tf with description, namespace, metric, stat, dims (a jsonencode(...)), period, evals, datapoints, op, threshold and missing. aws_cloudwatch_metric_alarm.this creates invdash-<key>, wires it to the ops intake topic (invdash-ops-alarms) and adds it to the ops dashboard's alarm widget.

C. New AWS event alert: add an entry to local.rules in infra/terraform/modules/alerting/main.tf with topic (ops or security), description and an EventBridge pattern. For an IAM or global sign-in event, put it in local.global_rules so us-east-1 forwards it.

Then: terraform fmt, terraform validate, terraform plan in infra/terraform, and deploy through the Deploy production workflow (its destroy gate checks the plan). New alarms and rules publish to the intake topics (invdash-ops-alarms / invdash-security-alarms), so the new alert goes through invdash-alert-notifier and is capped per issue and channel automatically (section 1). The issue key is the alarm name, or the event fields in How the issue key is derived. Never subscribe an inbox or phone to an intake topic or to invdash-ops-alerts-us-east-1.