Rotating secrets - alphagov/notifications-manuals GitHub Wiki

• API key for our apps to talk to our API
• API key for our apps to talk to document download API
• API key for our apps to talk to template preview

For our applications to talk our API they must use one of the API keys found in internal_client_api_keys in credentials/<env>/ssm/. If you want to rotate these keys then you should:

1. Add the new secret to credentials/<env>/ssm/internal_client_api_keys
2. Deploy the API so it now allows apps to auth with the new secret
3. Change e.g. credentials/staging/ssm/admin_client_secret to be the new secret
4. Deploy the affected apps e.g. document-download-frontend and the admin app
5. Remove the old secret from credentials/<env>/ssm/internal_client_api_keys
6. Deploy the API to preview and then run functional tests for all the other apps (antivirus, document download etc) to check they can still talk to the API. If so, you can then continue to deploy the API to production

For our applications to talk to the document download API they must use one of the API keys found in auth_tokens in credentials/<env>/ssm/. If you want to rotate these keys then you should:

1. Add the new secret to credentials/<env>/ssm/auth_tokens using a : separated list, for example secret1:secret2
2. Deploy the document download API so it now allows apps to auth with the new secret
3. Change credentials/<env>/ssm/document_download_api_key and DOCUMENT_DOWNLOAD_API_KEY in credentials/functional-tests/<env>-functional to be the new secret
4. Deploy the API
5. Remove the old secret from credentials/<env>/ssm/auth_tokens
6. Deploy the document download API to preview and then run functional tests for all the other apps (admin, api, document download, antivirus) to check they can still talk to the document download API. If so, you can then continue to deploy the document download API to production

For our applications to talk to the template preview they must use one of the API keys found in template_preview_internal_secrets in credentials/<env>/ssm. If you want to rotate these keys then you should:

1. Add the new secret to credentials/<env>/ssm/template_preview_internal_secrets
2. Deploy template preview so it now allows apps to auth with the new secret
3. Change template_preview_api_key in credentials/<env>/ssm
4. Deploy the API and the admin apps
5. Remove the old secret from credentials/<env>/ssm/template_preview_internal_secrets
6. Deploy template preview to preview and then run functional tests for the admin app and api to check they can still talk to template preview. If so, you can then continue to deploy template preview to production

This is achievable without downtime.

From the Logit stack's settings, go to the "Endpoints" page and hit "Add API Key" under "HTTPS API endpoint details":

Open a PR to notifications-credentials, updating credentials/<environment-name>/logit/api_key with the new api key for each affected environment (some Logit stacks have several environments sending logs to them) and get it merged.

Run scripts/upload-credentials/upload-credentials.sh (from the notifications-aws repo) against the affected environments (remember, these credentials are uploaded to the environment's aws account, not the notify-deploy account).

(Re-)run the deploy job of that environment's pipeline - the secret is read from the parameter store by terraform when it runs and provided to the lambda as a standard environment variable (no, this is not ideal).

After checking we're still getting app logs through to Logit, delete the old API key from the Logit control panel:

In theory this should be zero-downtime, but in practise switching the keys in logit doesn't always go smoothly. However, there shouldn't be any danger of permanent log loss even in the worst case scenario.

For an environment, somehow assume a working terraform shell (my favourite way is to fly hijack the deploy-infra-terraform task and manually initialise the shell by copy-pasting the first few commands of the task up to the terraform init). Then detach the currently provisioned access key from the terraform state using:

terraform state rm 'module.logit_user_cloudfront.aws_iam_access_key.iam_user_key["logit_cloudfront_<env-name>"]'

After this, re-run the deploy concourse job, during which terraform will provision a new access key to replace the detached one. The new key's credentials will be stored in the parameter store variables /notify/<env-name>/apps/logit/generated/logit_cloudfront_<env-name>/access_key and /notify/<env-name>/apps/logit/generated/logit_cloudfront_<env-name>/access_key. Set these values as the "Access Key ID" and "Secret Access Key" for all the S3 buckets configured as inputs for the environment's stack.

Once you're confident the stack is still receiving Cloudfront logs, can see the new access key being used in the AWS IAM console and the old access key not being used for some time (check this - do not take it for granted because sometimes logit needs to have its stack restarted or other manual intervention before it will successfully and exclusively be using the new key), delete the old access key manually.

Time-based one time passcodes (TOPTs) are generated using a seed key stored in the notifications-credentials repos. This is used to derive the OTP required to log into the SMS providers' UI. Seed keys must not be associated with a mobile authenticator app.

This section is written as if MFA does not exist on the account you're trying to log into. If it does then go to the Logging into the UI link here section for instructions on how to do this.

Log into Firetext, as per linked instructions, then click on the cog, then 2FA under My Settings.

The next steps vary depending on whether MFA already exists:

1a. MFA exists: Disable the existing MFA by entering your password and a 2FA Code and click Save Changes. This will immediately remove your MFA and cannot be undone.

1b. MFA does not exist: You can then enable 2FA, which will generate a QR code, containing the seed key. Take a screen grab of this seed key.

2. Go to https://zxing.org/w/decode.jspx and upload the picture of the QR code.

3. This will decode the image and give you the seed key and the backup

4. In notifications-credentials edit the notifications/firetext.gpg credential and replace the seed key and backup code (labelled OTP, Recovery key) for the appropriate env with the new key.

5. Commit and merge this change in.

This section assumes knowledge of the notifications-credentials repo, how to force ECS to do new deploys and that the rotation is for the production environment.

It does not talk about testing the key in different envs first, it is assumed sensible practice is being followed. The steps for rotating the key are applicable to all envs.

This API key can be rotated with zero downtime.

notifications-credentials/credentials/firetext.gpg contains the login details for all envs.

You will need to generate an OTP when logging into Firetext. To do this, decrypt the secret, which will give you something like this:

notify-pass show credentials/firetext                                                                      
# Other envs omiited for brevity

Development environments
------------------------
Email: notify-provider-accounts+firetext-dev@digital.cabinet-office.gov.uk
Password: Omitted
API key: Omitted
OTP: NRVWS2TVNB4WQZ3GOR4XK2TJNNXWY23KNBTXI6LIOVVGS23JOV4Q==== # Made up for this example
Recovery key: Omitted

Then generate a 6 digit OTP from the seed secret listed above as OTP

pip install pyotp
echo "<your OTP key here>" | python -c "import sys, pyotp; print(pyotp.TOTP(sys.stdin.read().strip()).now())" | pbcopy

e.g.

python -c 'import pyotp; print(pyotp.TOTP("NRVWS2TVNB4WQZ3GOR4XK2TJNNXWY23KNBTXI6LIOVVGS23JOV4Q====").now())' | pbcopy

# paste 526762

1. Go to settings > cog in top right corner > API > Add Key

You must then edit the existing secret in the notifications-credentials repo, using the instructions here and merge the change in. The credential can be found under the notifications-credentials/credentials/<env-name>/ssm/firetext_api_key path.

The ECS tasks then need rotating to force them to use the new API key which can be done in one of 2 ways:

1. (Recommended) a new deployment will rotate all existing workers and force them to use the new key.
2. Forcing a new deployment for the following services. AWS console > ECS > notify cluster >
   1. for dev envs api-worker, api-web
   2. for staging and prod:
   • api-web
   • api-worker-internal
   • api worker jobs
   • api worker letters
   • api-worker-periodic
   • api-worker-receipts
   • api-worker-report-requests-notifications
   • api-worker-reporting,
   • api-worker-research,
   • api-worker-retry-tasks
   • api-worker-sender
   • api-worker-sender-letters
   • api-worker-service-callbacks
   • api-worker-service-callbacks-retry
     
     

3. (Optional) You can check that the new task definition is being used by exec'ing onto the new container./

gds aws notify-<env name>-admin -- aws ecs execute-command --cluster notify --task <task-name>  --container <service name> --interactive --command '/bin/sh'

Then checking the env vars for the new API key

env | grep FIRETEXT_API_KEY

Once you are happy your new API key is being used successfully, you should initially deactivate (not delete) your old API key by clicking the cog icon next to the old API key, setting the status to off and clicking Save Changes.

After a period of at least a month of using your new API key with the old one deactivated you can safely delete the old key. Click on the bin icon of the disabled API key and click Remove Key. Deleted keys cannot be recovered.

1. Decrypt the MMG creds

cd path/to/notifications-credentials
pass credentials/mmg

2. Log into the relevant MMG account using the creds and go to the Services tab.

3. Under Create New API Connection: set:
   • The API Group Name: to the environment (Dev | Staging | Prod).
   • The API Account Type to Live.
   • The Delivery Receipt URL: to one of the following, depending on the env:
     • Dev - https://api.notify.works/notifications/sms/mmg
     • Staging - https://api.staging-notify.works/notifications/sms/mmg
     • Prod - https://api.notifications.service.gov.uk/notifications/sms/mmg
   • Click Create API Connection.

4. You will then be given your Username and Password for this API connection.

You must add them to the credentials/mmg credential now, as you will not be shown these details again. You must generate your API key using this info and add it to the same credential. You can do this by adding your integration name and password to the below code and running it.

echo -n 'integration-name:password' | openssl base64

e.g. 

echo -n 'MMG_cabo1721_2:e7awwbX2i5KjUhI' | openssl base64

TU1HX2NhYm8xNzIxXzI6ZTdhd3diWDJpNUtqVWhJ <-- This is the API key

5. You must then edit the existing secret in the notifications-credentials repo, using the instructions here and merge the change in. The credential can be found under the notifications-credentials/credentials/<env-name>/ssm/mmg_api_key path.

6. Raise a PR for the changes and merge them in.

The ECS tasks then need rotating to force them to use the new API key which can be done in one of 2 ways:

1. (Recommended) a new deployment will rotate all existing workers and force them to use the new key.
2. Forcing a new deployment for the following services. AWS console > ECS > notify cluster >
   1. for dev envs api-worker, api-web
   2. for staging and prod:
   • api-web
   • api-worker-internal
   • api worker jobs
   • api worker letters
   • api-worker-periodic
   • api-worker-receipts
   • api-worker-report-requests-notifications
   • api-worker-reporting,
   • api-worker-research,
   • api-worker-retry-tasks
   • api-worker-sender
   • api-worker-sender-letters
   • api-worker-service-callbacks
   • api-worker-service-callbacks-retry
     
     

3. (Optional) You can check that the new task definition is being used by exec'ing onto the new container.

gds aws notify-<env name>-admin -- aws ecs execute-command --cluster notify --task <task-name>  --container <service name> --interactive --command '/bin/sh'

Then checking the env vars for the new API key

env | grep MMG_API_KEY

Once you are happy your new API key is being used successfully, you should initially disable (not delete) your old API key by clicking Service > Modify on the key you want to disable > change API status to Disabled, click Update API.

After a period of at least a month of using your new API key with the old one disabled you can safely revoke the old key. Go to Service > Modify on the key you want to disable > click Revoke API.

Pagerduty (PD) has many 'integrations' which can trigger PD alerts (e.g. Concourse, Cronitor etc). As such there are multiple credentials associated with Pagerduty.

PD rotation is different to other credentials. You must be a PD admin to create new integration keys (the credential required for telling PD to trigger an alert). There is generally one PD admin per service area. At the time of writing this is Ishwar Pandeya. If they are absent, then communicate with Dean Wilson and failing that EE for help with the rotation to help with the rotation.

If you wish to check who is currently set as the PD admin then get a Github admin to go to this page and check.

Pagerduty's docs for rotating credentials are good. But please note that there is no option for creating multiple integration keys. Creating a new one will immediately overwrite the old one. Rotation cannot happen without some downtime.

There are around 20 PD credential entries in notifications-credentials, however these are not all unique entries. Below is a table stating which are shared, incidated by a matching letter in the Matching cred column.

 notify-pass credentials/maven-central/username 
 notify-pass credentials/maven-central/password 

 notify-pass edit credentials/maven-central/token-username 
 notify-pass edit credentials/maven-central/token-password 

1. Login to Maven Central using the credentials above.
2. Navigate to Team > View User Tokens.
3. You will see the token for Concourse. To rotate this, create a new token. Take note of the token username and token secret and update the notify-pass credential specified above.
4. Once the new token has merged into notifications-credentials you must revoke the old token in the Maven Central portal.

 notify-pass credentials/appveyor/username 
 notify-pass credentials/appveyor/password 
 notify-pass credentials/appveyor/2fa-key | python -c "import sys, pyotp; print(pyotp.TOTP(sys.stdin.read().strip()).now())"

 notify-pass edit credentials/appveyor/api_key  

1. log into the Appveyor portal.

2. Click on GOV.UK Notify in the top right hand corner > My profile.

3. Navigate to API Keys in the left hand menu.

4. Select GOVUKNotify in the drop down list of accounts to apply to. The current API key will display.

5. Click "generate new API token".

6. Update the token in notifications-credentials at the location above.

7. Once committed, revoke the old API token.

The API key that Appveyor uses to publish to Nuget may also be found in the Appveyor portal.

 notify-pass credentials/rubygems/email 
 notify-pass credentials/rubygems/password 
 notify-pass credentials/rubygems/otp | python -c "import sys, pyotp; print(pyotp.TOTP(sys.stdin.read().strip()).now())"

 notify-pass credentials/rubygems/api_key  

1. Login to the rubygems portal.

2. Under "My Gems" you will see the notifications-ruby-client.

3. Navigate to the profile icon drop down menu in top right and select "Settings".

4. Scroll down and select "API keys" to show the existing API key. You can create a new API key here.

5. Update notificiations-credentials at the location above.

6. Once the new API key is commited, revoke the old API key in the rubygems portal.

 notify-pass credentials/pypi/username 
 notify-pass credentials/pypi/password 
 notify-pass credentials/pypi/2fa-key | python -c "import sys, pyotp; print(pyotp.TOTP(sys.stdin.read().strip()).now())"

 notify-pass credentials/pypi/python-client-api-token  

1. Log into the PyPi portal. NB. email address is not accepted in the username field.

2. Once logged in navigate to top right corner "govuknotify" drop down list and select "Account Settings".

3. Scroll down to the "API tokens" section. Generate a new API token using the Add API token button.

4. Add the new token to notifications-credentials. Once it has been committed you must revoke the old token in the PyPI portal.

A separate wiki page has been written showing how to rotate the NPM client credentials. In this case the API key only has an expiry of 90 days. There is not currently a way to automate rotation of the API key, so we have chosen to do Just In Time rotation, as we only publish the client periodically.

The PHP client is not published to a remote repository for consumption. Instead users are directed to use a PHP dependency manager which targets our github repository. As such we do not have any API keys that need rotating for this client.