Deploying a Notify Development Environment - alphagov/notifications-manuals GitHub Wiki

This is an overview of the steps needed to configure and deploy a Notify development environment using the deploy pipeline. The intended audience is developers with existing knowledge of the Notify deploy pipeline, detailed here. An understand of AWS' Parameter Store and how to log into AWS via the GDS CLI.

• Pull the latest version of these three repos:

  • notifications-credentials
  • notifications-aws
  • notifications-aws-account-wide-terraform

• You will need an up-to-date version of:

  • GDS-CLI
  • Terraform using tfenv (v1.7.5 and above), instructions for macOS.

• Find a development environment that is not in use by checking the #govuk-notify-infrastructure-team slack channel topic

• Update the #govuk-notify-infrastructure-team Slack channel topic to claim the development environment you are going to use.

• Make sure you are on the VPN (you will need this for concourse and set-pipeline-vars)

• Check you have the $PASSWORD_STORE_DIR environment variable set by running echo $PASSWORD_STORE_DIR, pointing to the local directory containing notifications-credentials. If you do not have this set, add this line to your ~/.zshrc file and load a new terminal window:

• All actions on this page are run against the notify-deploy AWS account.

# ~/.zshrc
export PASSWORD_STORE_DIR=/full/path/to/notifications-credentials

From your terminal navigate to your notifications-aws-account-wide-terraform repo directory. Run the following command where <env> is formatted like "dev-c":

gds aws notify-<env>-admin -- make <env> apply

In most cases it is unlikely to make any changes, but is worth checking before applying.

From your terminal navigate to your notifications-aws/scripts/upload-credentials dir. Run the following command, where <env> is formatted like "dev-c":

AWS_PAGER="" gds aws notify-<env>-admin -- ./upload-credentials.sh <env>

In most cases, there is no output if successful and will upload dev environment-specific credentials needed for the deployment. These update the parameter store of eu-west-1 in the specified account.

An exception to this rule is that you may see some errors relating to missing items in the password store. This is normal, as these are creds used by staging and production.

In concourse each development environment has a concourse team and within that team a concourse pipeline called deploy-notify. This makes up the deployment pipeline for that whole environment. The pipeline consists of four tabs:

• deploy
• operator
• pack-bag
• destroy

To deploy an environment you initially run the pack-bag job. Select the deploy job, then trigger new build. Once complete, this triggers the start-deploy job in the deploy tab of the deploy-notify pipeline.

To destroy an environment you can run the destroy job. When you do, don't forget to update the #govuk-notify-infrastructure-team slack channel topic to show the dev environment is free for the next person.

To perform pipeline control tasks you can use jobs in the operator tab, for example if you need to forcibly release the pipeline lock if it has not been relinquished by a failed job (care must be taken to ensure no jobs are running that have legitimately taken the lock).

The deploy-notify pipeline is a single pipeline for the whole of Notify. The versions of the different apps which make up a deployment are dictated by the resources that are input to the pack-bag job.

You should initially pin the versions of each resource while you are developing a feature, as this ensures that changes to the latest versions of other apps do not interfere with the effects of changes you are working on.

Each resource input to the pack-bag job is sourced from an AWS Parameter Store (SSM) variable, specifically in the notify-deploy AWS account in the London region eu-west-2.

To quickly pin all resources in an environment:

fly --target notify-dev-<env> login --team-name dev-<env> --concourse-url https://concourse.notify.tools/ -b
fly -t notify-dev-<env> login -b  

cd path/to/notifications-aws/concourse/deploy-notify
make <env> pin-bag-resources

1. In your Terminal, navigate to your notifications-aws/concourse/deploy-notify directory.
2. Use the following make target, where <env> is formatted like "dev-c":

AWS_PAGER="" gds aws notify-deploy-devsecrets -- make <env> set-pipeline-vars

There may be some errors relating to missing items in the password store. This is normal, as these are creds used by other builds.

There are 2 resource types you can use as pack bag inputs:

1. Docker resources - the apps, stubs and images that make up Notify.

2. Git resources - Code stored in a git repo; in our case this is only notifications-aws.

gds aws notify-deploy-devsecrets -l

There are 2 ways:

1. Make the changes to your branch, push the commit(s) and create a PR.
2. Go to eu-west-2's parameter store and find the parameter /concourse/concourse/pipelines/dev-<a-f>/deploy-notify/notifications_aws_repo_branch. Change its value from main to the name of your branch.
3. Unpin the notifications-aws resource. This resource will be pulled from the branch you set in parameter store.
4. Manually trigger the packbag resource. This will add the update resource to the packbag.
5. This should then cause the rest of the pipeline to auto apply.

In the console this looks like

1. Make the changes to the resources in notifications-aws you want to.
2. cd path/to/concourse/deploy-notify/deploy-notify.yml.j2
3. Remove the render-pipeline job. This will remove this job from the deployment pipeline when you run it. Necessary as that job would apply the pipeline config which is set in the branch you've set in parameter store (main by default) - not your local pipeline file's config.
4. Manually update the pipeline make dev-<a-f> set-pipeline.
5. Check your changes and apply if happy apply them
6. Trigger a new build of the start-deploy job. You should see the render-pipeline job has disappeared.

Docker resources refer to the apps, stubs and images that make up Notify.

When working on a branch, changes to docker resources are built into a 'demo image' which allow the changes to be tested before merged into main. Once merged, docker resources are built into 'final images' in the final-images-builds Team in Concourse.

When demo images are built they are pushed toeu-west-1's ECR under the resource name with -demo appended e.g. if you want to test changes to the notifications-api docker resource, then the demo image will be pushed to the notifications-api-demo ECR repo.

They are pushed with a tag in the following format pr-<pr-number>-production-YYYY-MM-DD-HH-MM-SS-<commit-SHA> e.g.
pr-4524-production-2025-07-30-13.57.09-5195542d0250201cfdbc9f5b5a576964aaa8c05a

Tag for final images are in the format YYYY-MM-DD-HH-MM-SS-<commit-SHA>

Parameters will be in this format and both must be updated

• /concourse/concourse/pipelines/<env-name>/deploy-notify/<docker-resource-name>_repo_name e.g.
  /concourse/concourse/pipelines/dev-d/deploy-notify/notifications_api_repo_name

and

• /concourse/concourse/pipelines/<env-name>/deploy-notify/<docker-resource-name>_repo_tag_regex e.g.
  /concourse/concourse/pipelines/dev-d/deploy-notify/notifications_api_repo_tag_regex

For all images which match the _tag_regex, the latest version is used.

1. Go to eu-west-2's parameter store.
2. Find the _repo_name parameter for your docker resource and change the value to append -demo e.g. notifications-api becomes notifications-api-demo.
3. Find the tag_regex parameter for your docker resource and change the value to ^pr-<PR number>-.* where PR number is the number of your PR which triggered the build of the demo image e.g. ^pr-4532-.*
4. Unpin the docker resource and you should see an image which matches the repo_name and tag_regex
5. Manually run the pack bag job which should trigger a deploy of the new image.

You can confirm your new image will be deployed by seeing an update to the task_definition in the deploy job, deploy-infra-terraform task.

In this console this look like

You can ignore this error, the devsecrets account has limited permissions intentionally:

Return to concourse and select the team for your dev environment. In the pack-bag tab you can now click into that resource and verify that the latest hash of the commit (if a git-repo resource) or hash of the target image in ECR is correct. It may take a few minutes for concourse to detect changes to its SSM source variables and retrieve the correct resources:

Once you have set up the required inputs to your pack-bag and pinned the versions of every resource that you are not working on, you can start the pack-bag job to begin a deployment of the dev environment.

As the deployment progresses through the jobs of the deploy pipeline the same deployment-bag is fetched by each job, assuring that the same versions of each app are played together. You must allow the jobs to run in sequence before you can assume that the latest deployment is available (e.g. don't force the tests job to run before the deploy job has completed).

If a job fails for transient reasons you can rerun it by clicking into the job and clicking "trigger a new build". Because of the way the deployment-bag is consumed, it will run idempotently as long as the same deployment-bag is still available (if you have modified variables in SSM since deployment you may find old deployment bags are no longer available).

Resources should be unpinned in your env when you have finished.

To quickly pin all resources in an environment:

fly --target notify-dev-<env> login --team-name dev-<env> --concourse-url https://concourse.notify.tools/ -b
fly -t notify-dev-<env> login -b  

cd path/to/notifications-aws/concourse/deploy-notify
make <env> unpin-bag-resources

You have now set up your dev env 🎉 . You can now access your custom Notify app at:

https://www.<env>.notify-<env>.space

Each time a dev env is brought build anew (not woken) all services within that env must be created. This can only be done via the console.

Important note. The instructions in this sub section do not apply to dev-livemail. In dev-livemail SES is in production mode and can send emails to any address.

The Simple Email Service (SES) in the dev environments is rate-limited and running in sandbox mode meaning you can only send email to verified identities. This is an email addresses that you've proven that you own.

Therefore to sign up for a Notify account or send email notifications to yourself via Notify in a dev environment, you must make sure your email address is configured as a verified identity in SES.

SES sandbox identities are not managed via terraform, and should be manually added to each dev environment that you are using this is a one-time operation. To do this, log in to the dev environment

gds aws notify-<env>-admin -l

From the AWS console, choose Amazon Simple Email Service (SES) using the services search menu. Then click Identities under the Configuration menu on the left hand menu. Here you will see a list of identities set up in SES.

Use the "Create Identify" button to add a new email address (not domain) to the list that SES is allowed to send to. SES will send a verification message with a magic link to the email address that you configured to verify you are able to access the address. Once an identity is verified, the dev environment will be able to send emails to that address.

You can also do this with the CLI

gds aws notify-dev-<>-admin -s  # open a subshell
aws ses verify-email-identity --email-address <email address> --region eu-west-1
exit  # exit the subshell

The pipeline itself is defined as a jinja2 template deploy-notify.yml.j2 at notifications-aws/concourse/deploy-notify.

The template is built using a script called render-pipeline.sh.

You can make amendments to how the pipeline is constructed by setting Jinja2 variables then running the set-pipeline make target. You can see what Jinja2 variables are available by looking in the render-pipeline.sh script.

To do this firstly login to concourse using the fly CLI tool where <env> is like "dev-c".

fly -t notify-<env> login

Then set a Jinja variable before running the make target from the same directory (notifications-aws/concourse/deploy-notify):

JINJA_BOOL_include_continuity_tests=1 make <env> set-pipeline

Viewing in concourse you will now see the continuity test job appears as part of the pipeline:

"Shutting down" currently means destroying. This can be done by going to the destroy tab of the pipeline and triggering a new "start-destroy" job. Anything you have not deployed to this env via Terraform (i.e. notifications-aws) will be lost. Other jobs need to be finished before running destroy (it shares the pipeline lock).

After the destroy is complete go back to the pack bag and unpin the components (so the environment is ready for the next person to configure).

Don't forget to edit the slack channel topic on #govuk-notify-infrastructure-team to show that the dev environment that has been destroyed is now available for reuse.

At 1900 each night the dev envs are put to 'sleep' to cost save. You can manually put your env to sleep by going to the operator tab > sleep-environment > trigger a new build.

Your env will not be usable once sleeping.

Sleeping envs are not automatically woken the next day. You must manually do this by going to the operator tab > wake-environment > trigger a new build.

If you find that the deployment seems to hang at the start-deploy job, specifically on the step acquiring the pipeline-lock, you may need to forcibly unlock the pipeline using the force-unlock-pipeline job under the operator tab (⚠️take care that no jobs are running that have legitimately taken the lock).

This could be because of a time out, or something being wrong in the dependency graph. You should initially re run the job with the same inputs.

This is an intermittent bug and the job should be re run with the same inputs.

This means there are no changes to the packbag

Your email address may not be registered with SES. See Creating accounts and sending notifications from your dev environment section.

Documented in detail here, indicates you're trying to deploy an image which no longer exists in ECR.

This is likely due to the pipeline vars being deleted. You should try re setting the pipeline vars by following these instructions after pulling an up-to-date version of the notifications-credentials repo down.

cd ./notifications-credentials
git checkout main
git fetch
git reset --hard FETCH_HEAD

If the pack bag job is not completing then it is likely your env inputs in SSM parameter store are incorrect. The packbag inputs, for e.g. notifications-api, may be incorrect, either, or both, the tag regex, or the branch name. Follow the instructions here to set the correct inputs.