This runbook covers the scheduled rotation of secrets in the tmi-secrets Kubernetes Secret (tmi#965), available from server 1.17.0. A CronJob named tmi-rotator runs daily and rotates the Redis password and the settings encryption key. Each run does at most one idempotent step per rotation, so a rotation that needs a server rollout finishes over one or more runs and resumes after any failure.

All commands use namespace tmi-platform. On AWS, add --context <your-context> to every kubectl call. Never print Secret data; read annotations only.

• The rotator writes tmi-secrets and waits for tmi-server to roll (Stakater Reloader rolls it when the Secret data changes; annotation-only changes do not roll it).
• The CronJob schedule is 17 3 * * * (daily). concurrencyPolicy: Forbid, backoffLimit: 0, so a failed phase is resumed by the next day's run, not retried in a loop.
• The rotator runs settings-key first, then redis-password. A failure in one does not stop the other; the Job exits 1 if any failed.
• Redis is not restarted for a password rotation. The rotator changes the password with ACL SETUSER and the new value is written to the Secret. A restarted Redis takes its password from tmi-secrets.

A rotation is started when it is never-rotated (no rotated-at annotation), when rotated-at is unreadable, or when rotated-at is older than its interval. A rotation whose phase annotation is set is always resumed, whether or not it is due.

Set the annotation tmi.dev/rotate-every.<name> on tmi-secrets. The value is either <n>d (days, 1 to 36500) or any Go duration such as 720h. Zero, negative, or unparsable values are logged as a warning and the 90d default is used (never "rotate now").

kubectl -n tmi-platform annotate secret tmi-secrets tmi.dev/rotate-every.redis-password=30d --overwrite

Phase is stored in tmi.dev/rotation-phase.<name>. No annotation means idle.

redis-password

settings-key

make rotate-secret CLUSTER=docker-desktop name=redis-password
make rotate-secret CLUSTER=k3s name=settings-key
make rotate-secret CLUSTER=aws CONTEXT=<your-context> name=settings-key

name is redis-password or settings-key. CONTEXT is required for aws and is never taken from your current context. The command creates a Job from the CronJob, sets ROTATE=<name>, waits up to 40 minutes, and prints the Job logs. Forced Jobs expire 24 hours after they finish (ttlSecondsAfterFinished), so read their logs before then.

A forced run starts the named rotation regardless of age and also performs any other rotation that is due or in progress. A secret that has never been rotated is due on its first run, so the first forced or scheduled run rotates both.

Note: the rotator binary itself uses exit 1 for a failed rotation and exit 2 for configuration or connection errors (Kubernetes, Redis, database).

Annotations only:

kubectl -n tmi-platform get secret tmi-secrets -o json | jq '.metadata.annotations | with_entries(select(.key | startswith("tmi.dev/")))'

Recent Jobs and logs:

kubectl -n tmi-platform get cronjob tmi-rotator
kubectl -n tmi-platform get jobs -l app=tmi-rotator
kubectl -n tmi-platform logs job/<job-name>

Every run logs one rotation status line per rotation before deciding what to do, with rotation (the rotation name), age_days (whole days since rotated-at, or -1 if never rotated) and phase. Other useful lines: Rotation due, Rotation forced, Rotation resuming, Rotation phase recorded, Rotation failed, Rotation run complete.

Phases are resumable: fix the cause, then re-run (make rotate-secret ... or wait for the next CronJob run). Do not clear a phase annotation unless a case below says so.

Stuck phase. Read the Rotation failed line in the last Job log. Common causes: tmi-server did not finish rolling within TMI_ROTATOR_ROLLOUT_TIMEOUT (default 10m; fix the rollout, then re-run), the database or Redis was unreachable, or, on AWS, the settings-key escrow write failed (the rotation stays in staged until a run can write it).

Lost tmi-secrets or namespace (AWS). Restore the settings key from escrow; see Restoring the settings key from escrow.

"another rotator run is active?" (conflict). The phase changed between the rotator's read and write, meaning a second run overlapped or someone edited the annotation. Overlapping forced runs behave this way: one exits with the conflict (also reported as "secret changed since it was read") and writes nothing, and the other completes. Check kubectl get jobs -l app=tmi-rotator for a running Job and re-run once it ends.

Abandoning a redis-password rotation in swapped. Nothing needs undoing: the Secret already holds the working password and Redis accepts it. Remove the phase annotation to leave the rotation idle:

kubectl -n tmi-platform annotate secret tmi-secrets tmi.dev/rotation-phase.redis-password-

The old password stays valid in Redis until the next Redis restart (Redis rebuilds the default user from the Secret at every start) or the next completed Redis rotation, whose retire step drops it.

Redis rejected the password in tmi-secrets. The rotator logs "Redis rejected the password in tmi-secrets" and exits 2. This happens if Redis restarted between the Secret write and the roll. Redis takes its password from the Secret at start, so:

kubectl -n tmi-platform rollout restart deploy/redis

The append-only file keeps data across the restart (sessions survive). Then re-run the rotator. The same command fixes a tmi-server that logs WRONGPASS: the restart reloads the password from the Secret.

Redis restart during a Redis rotation. Redis takes its password from tmi-secrets when its pod starts, and the rotator re-reads the password from the Secret each time it connects or reconnects. A Redis restart after the rotator swaps the Secret to the new password is harmless: Redis starts with the new password and the run completes. The bad case is a restart in the short window after the rotator adds the new password to Redis (in memory only) but before it swaps the Secret. Redis then starts with only the old password, but the Secret, tmi-server and the rotator all move to the new one, which Redis rejects. Later rotator runs cannot fix this. tmi-server cannot reach Redis until Redis restarts again and loads the new password; recover with the rollout restart deploy/redis command under "Redis rejected the password in tmi-secrets" above.

Orphaned Redis passwords. If a run adds the new password to Redis and then loses the compare-and-swap on the Secret (see the conflict case), that password stays accepted by Redis but is not in tmi-secrets. No manual ACL cleanup is needed: when a rotation completes, the retire step (ACL SETUSER default resetpass >NEW) drops every password except the one in the Secret. A Redis restart also discards them.

Settings key refused to promote. staged phase but the staged key pair is missing or inconsistent; refusing to promote means _PREVIOUS_KEY is empty, or _PREVIOUS_CONTEXT_ID is not the current context id plus one, while the phase is staged. In staged the rotator has only added the previous pair; the current key and id are unchanged and nothing is encrypted under the staged key. Two options, both supported by the code:

Untested. These recovery steps were derived from reading the rotator code and have not been exercised on a real cluster. Read the annotations first and proceed carefully.

1. Restore the pair: put a 64-hex-character key in TMI_SECRET_SETTINGS_ENCRYPTION_PREVIOUS_KEY and set TMI_SECRET_SETTINGS_ENCRYPTION_PREVIOUS_CONTEXT_ID to the current TMI_SECRET_SETTINGS_ENCRYPTION_CONTEXT_ID plus one (if the current context id is absent, it counts as 1, so use 2). Re-run; the rotator promotes it.
2. Start over: remove the phase annotation and both previous entries. With no phase and no previous id, the next due or forced run stages a fresh key. Removing the phase alone does not work: a run refuses to start while a previous key id is present.

kubectl -n tmi-platform annotate secret tmi-secrets tmi.dev/rotation-phase.settings-key-

Remove the two previous entries with kubectl patch --patch-file (a umask 077 file setting each to null). Use either option only in staged, and only if the current key and id were never edited. In promoted or reencrypted, or if the current key or id changed, the new key may already encrypt rows: contact the maintainers.

_PREVIOUS_KEY without _PREVIOUS_CONTEXT_ID. An existing deployment with TMI_SECRET_SETTINGS_ENCRYPTION_PREVIOUS_KEY but no ..._PREVIOUS_CONTEXT_ID makes the CronJob fail every day by design: the rotator refuses to guess the id. Add the id (a positive integer, the id the previous key was current under, normally 1), or remove the previous key if nothing is encrypted under it. Also: a run refuses to start a new settings-key rotation while any previous key id is present.

Undecryptable settings rows. Rows the rotator cannot re-encrypt are logged as Setting could not be re-encrypted key=<setting key> (key only, never values) and skipped. They do not fail the run, but the previous key is not dropped while any row still uses its id. Investigate or delete those settings, then the next run drops the key.

After re-encryption, the old settings key is kept for TMI_ROTATOR_SETTINGS_PREVIOUS_GRACE (Go duration, default 192h, 8 days) because encrypted session and refresh-token entries in Redis may still be encrypted under the old key id until their TTLs expire. The default assumes a 7-day maximum lifetime. If auth.jwt.refresh_token_days or auth.jwt.session_lifetime_days exceed 7, raise the grace to at least that many days plus one day. Set it as a container env var on the tmi-rotator CronJob. Other rotator settings are TMI_ROTATOR_NAMESPACE, TMI_ROTATOR_SECRET, TMI_ROTATOR_SERVER_DEPLOYMENT, TMI_ROTATOR_ROLLOUT_TIMEOUT, TMI_ROTATOR_SETTINGS_ESCROW_SECRET_ARN (see Configuration Reference).

Two alarms notify the security alerts topic: tmi-secret-rotation-failing catches a failed run within a day, and tmi-secret-rotation-stale catches a secret that has gone too long without rotating.

Every rotator run that fails logs "Rotation run failed: ...". A log metric filter counts these lines (TMI/Rotator RotationRunFailed), and the alarm fires when there is at least one in a day. Without it, a rotation that fails every night would go unnoticed until the stale alarm fires at 100 days. Days with no failures produce no datapoints, which counts as healthy. When it fires, read the newest Job log.

The rotator's rotation status log line feeds a metric (TMI/Rotator SecretAgeDays, taken from age_days). The alarm tmi-secret-rotation-stale fires when the daily maximum exceeds 100 days for two consecutive days, and notifies the security alerts topic.

• Threshold: 100 days (the default interval is 90d, leaving a 10 day margin).
• Missing data counts as breaching. Two consecutive days with no datapoint means the CronJob is not running (suspended, image pull failure, cluster down); check kubectl get cronjob tmi-rotator and recent Jobs.
• A never-rotated secret reports age -1, which never breaches. A fresh cluster still alarms after two days until the rotator's first run has produced a datapoint.
• An alarm with a healthy CronJob means a rotation is failing repeatedly: read the newest Job log.

Terraform seeds tmi-secrets once, then ignores its data and annotations (ignore_changes). The rotator is the only writer of the rotated keys.

• To change a value Terraform seeded but the rotator does not rotate (database URL, JWT secret), patch the Secret manually with a umask 077 patch file so the value never reaches argv or history: kubectl -n tmi-platform patch secret tmi-secrets --patch-file <file>, then delete the file.
• Never terraform apply -replace, taint, or delete tmi-secrets or its namespace while RDS holds encrypted settings. Terraform would reseed the original settings key, and rows encrypted under a rotated key become unreadable.
• Since server 1.18.0, the rotated settings key has an off-cluster copy in Secrets Manager (see Settings-key escrow (AWS)). The escrow covers only the settings key: recreating tmi-secrets still resets every other rotated value, such as the Redis password, to its seed.

Since server 1.18.0, the rotator on AWS copies the settings key to AWS Secrets Manager before each promotion (tmi#1009). Without that copy, losing tmi-secrets or the tmi-platform namespace after a rotation would make every encrypted (ENC:) settings row permanently unreadable.

• Where: the Secrets Manager secret tmi-settings-key-escrow, created by the aws-persistent Terraform stack with prevent_destroy, so destroying a deployment does not delete it. Apply aws-persistent before aws-public: aws-public looks the secret up by name and fails if it does not exist.
• Configuration: the rotator reads the secret's ARN from TMI_ROTATOR_SETTINGS_ESCROW_SECRET_ARN, which the tmi-rotator-config ConfigMap supplies. If it is unset, as on the docker-desktop and k3s dev clusters, there is no escrow.
• Access: on AWS the CronJob runs as the ServiceAccount tmi-rotator-aws, whose IRSA role may only call secretsmanager:PutSecretValue on that one secret.
• When: in the staged phase, after the server has rolled and the key pair has been checked, and strictly before the move to promoted. The payload holds the post-promotion pair: current is the new key and previous is the key it replaces. A rerun after a failure writes the escrow again, which only adds a version.
• Failure: if the escrow write fails, the rotation does not promote and stays in staged (it fails closed). The run fails, so tmi-secret-rotation-failing fires, and if it keeps failing, so does tmi-secret-rotation-stale.
• Terraform creates no version of the secret. The secret has no value until the first settings-key rotation reaches staged.

Status: Untested. This runbook has not been exercised end to end. Rehearse it against a non-production cluster before relying on it.

Use this when the tmi-secrets Kubernetes Secret (or the tmi-platform namespace) is lost and the settings-encryption key must be recovered, or when a rotation left the Secret inconsistent. The escrow secret is tmi-settings-key-escrow in the deployment's AWS account and region. The commands below use the tmi profile and us-east-1, the deploy-aws.sh defaults; change them to match your deployment. As elsewhere on this page, add --context <your-context> to every kubectl call. Background: ADR: settings-key escrow.

Never print the payload or the key. Do not cat, echo or paste these files anywhere. Run this in your own terminal. It aborts on any failure and validates every file; files land in a mode-700 directory with mode-600 permissions:

(
  set -euo pipefail
  umask 077
  d=$(mktemp -d)
  wipe() { rm -P "$@" 2>/dev/null || shred -u "$@" 2>/dev/null || rm -f "$@"; }
  trap 'rc=$?; if [ $rc -ne 0 ]; then wipe "${d:?}"/*; rmdir "${d:?}"; else wipe "${d:?}/escrow.json"; fi' EXIT
  aws secretsmanager get-secret-value --secret-id tmi-settings-key-escrow --profile tmi --region us-east-1 --query SecretString --output text > "$d/escrow.json"
  jq -er '.current.key_hex'  "$d/escrow.json" | tr -d '\n' > "$d/TMI_SECRET_SETTINGS_ENCRYPTION_KEY"
  jq -er '.current.id'       "$d/escrow.json" | tr -d '\n' > "$d/TMI_SECRET_SETTINGS_ENCRYPTION_CONTEXT_ID"
  jq -er '.previous.key_hex' "$d/escrow.json" | tr -d '\n' > "$d/TMI_SECRET_SETTINGS_ENCRYPTION_PREVIOUS_KEY"
  jq -er '.previous.id'      "$d/escrow.json" | tr -d '\n' > "$d/TMI_SECRET_SETTINGS_ENCRYPTION_PREVIOUS_CONTEXT_ID"
  for f in TMI_SECRET_SETTINGS_ENCRYPTION_KEY TMI_SECRET_SETTINGS_ENCRYPTION_PREVIOUS_KEY; do
    [[ $(wc -c < "$d/$f") -eq 64 ]] && LC_ALL=C grep -Eq '^[0-9a-fA-F]{64}$' "$d/$f" || { echo "ABORT: $f is not 64 hex chars" >&2; exit 1; }
  done
  for f in TMI_SECRET_SETTINGS_ENCRYPTION_CONTEXT_ID TMI_SECRET_SETTINGS_ENCRYPTION_PREVIOUS_CONTEXT_ID; do
    LC_ALL=C grep -Eq '^[1-9][0-9]*$' "$d/$f" || { echo "ABORT: $f is not a positive integer" >&2; exit 1; }
  done
  echo "files in: $d"   # prints only the directory name
)

If it aborts, the trap has already deleted the directory and every file in it; do not continue. On success only the four value files remain, and escrow.json is removed. Afterwards run d=<the printed path> in the same terminal before steps 3 and 4.

The payload is {"rotation":"settings-key","escrowed_at":...,"current":{"id","key_hex"},"previous":{"id","key_hex"}}. The roles are post-promotion: current is the key the server should use now, previous is the key it replaced. The rotator always writes a previous entry (it is the keyring's old current key), so all four values exist.

The ids are JSON integers in the escrow; jq writes them as their decimal string, which is the form the Secret needs. The server (internal/crypto/settings_encryptor.go) and the rotator (internal/rotator/settings_key.go) both parse the value with strconv.Atoi and require it to be greater than 0. The key values are 64 hex characters.

Never pass a value in argv or an environment variable; use the files from step 1 ($d).

Preferred: patch the existing Secret. Merge only these four keys so everything else is untouched. Optionally scale the server to zero first (kubectl -n tmi-platform scale deployment/tmi-server --replicas=0, restore the previous replica count afterwards) so nothing writes settings with the wrong key meanwhile.

kubectl create secret generic x --from-file="${d:?}" --dry-run=client -o json \
  | jq '{data: .data}' \
  | kubectl -n tmi-platform patch secret tmi-secrets --type=merge --patch-file=/dev/stdin

The jq filter drops the throwaway name and metadata so the patch carries only the four base64 data entries; they travel through the pipe, never argv.

If tmi-secrets is gone. Recreating it from only these four keys is incomplete: the server also needs the database URL, JWT secret and Redis password, among other keys. On AWS the Secret is owned by Terraform (kubernetes_secret_v1.tmi in terraform/modules/kubernetes/aws/k8s_resources.tf, applied through scripts/deploy-aws.sh, which also applies the server). Re-run AWS_PROFILE=tmi make deploy-aws to recreate the Secret with its seed values. That seed contains the original settings key from Terraform variables, NOT the rotated one, and the server may start with it. So right after the deploy, scale tmi-server to 0, apply the patch above, then scale it back up. Rotated Redis and other values are also reset to seeds by recreation and need their own recovery.

Afterwards remove the files securely (rm -P "${d:?}"/* 2>/dev/null || shred -u "${d:?}"/*, then rmdir "${d:?}") and roll the server: kubectl -n tmi-platform rollout restart deployment/tmi-server.

Rotation progress lives in annotations on tmi-secrets, suffixed .settings-key. A restore always puts a PREVIOUS key and id into the Secret. With no phase annotation the rotator refuses every run ("previous key id N still present; refusing to start a new rotation"), so pick one of these:

Option A (preferred): let the normal drop logic clean up. Set the phase to reencrypted and stamp promoted-at with the current time (RFC3339 UTC). After the previous-key grace period (default 192h) the rotator drops the previous key once no rows use the old id:

now=$(date -u +%Y-%m-%dT%H:%M:%SZ)
kubectl -n tmi-platform annotate secret tmi-secrets --overwrite \
  tmi.dev/rotation-phase.settings-key=reencrypted \
  tmi.dev/rotation-promoted-at.settings-key="$now" \
  tmi.dev/rotated-at.settings-key="$now"

Option B: remove the previous key yourself. Only once you have confirmed no settings rows still use the previous id:

kubectl -n tmi-platform patch secret tmi-secrets --type=json -p '[
 {"op":"remove","path":"/data/TMI_SECRET_SETTINGS_ENCRYPTION_PREVIOUS_KEY"},
 {"op":"remove","path":"/data/TMI_SECRET_SETTINGS_ENCRYPTION_PREVIOUS_CONTEXT_ID"}]'
kubectl -n tmi-platform annotate secret tmi-secrets tmi.dev/rotation-phase.settings-key- tmi.dev/rotation-promoted-at.settings-key-
kubectl -n tmi-platform annotate secret tmi-secrets --overwrite tmi.dev/rotated-at.settings-key="$(date -u +%Y-%m-%dT%H:%M:%SZ)"

Other annotations: tmi.dev/rotate-every.settings-key (default 90 days) with rotated-at decides when the next rotation starts (a missing or unreadable rotated-at counts as due); tmi.dev/rotation-generation.settings-key is bookkeeping for the rollout check, leave it alone. If the Secret still carries a stale staged or promoted phase from before the loss, replace it with reencrypted as in Option A only when the restored pair is consistent with the database rows; if you are unsure, leave the phase alone until you have confirmed which key id the rows use.

Each rotation writes a new version of the escrow secret. The latest is AWSCURRENT and the one before it is AWSPREVIOUS. Secrets Manager deprecates versions beyond the latest two, so older versions are not reliably recoverable. To restore from the prior version, add --version-stage AWSPREVIOUS to the get-secret-value command in step 1; the same no-display handling applies.

Redis stores an append-only file on the redis-data PVC, so sessions and other encrypted entries survive Redis restarts and rolls. Sessions are encrypted with the settings key held in tmi-secrets, so the two must stay together.

• make dev-down keeps both the PVC and tmi-secrets.
• make dev-nuke deletes the namespace, removing tmi-secrets and the PVC together, and the next start reseeds a fresh password.
• Redis runs with --maxmemory 384mb --maxmemory-policy noeviction under a 640Mi container limit. That cap keeps an append-only-file rewrite inside the 1Gi PVC. With noeviction, Redis never drops keys, so a revoked token cannot become valid again through eviction. At the cap Redis refuses writes instead. Token revocation (POST /oauth2/revoke) and client-credential deletion (DELETE /me/client_credentials/{id}) then fail closed with 503 and Retry-After, and nothing is deleted, so the call can be retried. Every key has a TTL and the working set is small, so this is unlikely.
• redis-data survives make dev-down and make dev-up.
• Never delete one without the other: a PVC without its Secret holds entries no key can read, and a Secret without its PVC loses sessions.

Only when needed; authentication comes from the pod's own environment, so no password reaches your terminal:

kubectl -n tmi-platform exec deploy/redis -- sh -c 'REDISCLI_AUTH="$REDIS_PASSWORD" redis-cli --tls --cacert /tls/ca.crt <cmd>'

These are planned and not yet operational; this page will be extended as each lands.

• JWT signing keyring (ES256).
• Database user passwords.
• Internal CA rotation (trust old and new CA, reissue leaf certificates, drop the old CA).