This is an overview of the steps needed to configure and deploy a Notify development environment using the deploy pipeline. The intended audience is developers with existing knowledge of the Notify deploy pipeline, detailed here.

Before building a development environment you should:

• Find a development environment that is not in use by checking the #govuk-notify-infrastructure-team slack channel topic
• Update the #govuk-notify-infrastructure-team slack channel topic to claim the development environment you are going to use.
• Make sure you are on the VPN (you will need this for concourse and set-pipeline-vars)
• Pull the latest version of these three repos:
  • notifications-credentials
  • notifications-aws
  • notifications-aws-account-wide-terraform
• Check you have the $PASSWORD_STORE_DIR environment variable set by running echo $PASSWORD_STORE_DIR. This should point to the local directory containing notifications-credentials. If you do not have this set, add this line to your ~/.zshrc file and load a new terminal window:

export PASSWORD_STORE_DIR=/full/path/to/notifications-credentials

• You will need an up-to-date version of:
  • GDS-CLI
  • Terraform using tfenv (v1.7.5 and above), instructions for macOS.

From your terminal navigate to your notifications-aws-account-wide-terraform repo directory. Run the following command where <env> is formatted like "dev-c":

gds aws notify-<env>-admin -- make <env> apply

This will deploy the latest account-wide terraform to the dev environment updating things like concourse worker permissions. In most cases it is unlikely to make any changes, but is worth checking before proceeding.

From your terminal navigate to your notifications-aws/scripts/upload-credentials repo directory. Run the following command, where <env> is formatted like "dev-c":

gds aws notify-<env>-admin -- ./upload-credentials.sh <env>

This will upload dev environment specific credentials needed for the deployment. These update the parameter store of eu-west-1 (Ireland) in the dev specific account.

In concourse each development environment has a concourse team and within that team a concourse pipeline called deploy-notify. This makes up the deployment pipeline for that whole environment. The pipeline consists of four tabs:

• deploy
• operator
• pack-bag
• destroy

To deploy an environment you initially run the pack-bag job, which once complete, triggers the start-deploy job in the deploy tab of the deploy-notify pipeline.

To destroy an environment you can run the destroy job. When you do, don't forget to update the #govuk-notify-infrastructure-team slack channel topic to show the dev environment is free for the next person.

To perform pipeline control tasks you can use jobs in the operator tab, for example if you need to forcibly release the pipeline lock if it has not been relinquished by a failed job (care must be taken to ensure no jobs are running that have legitimately taken the lock).

The deploy pipeline is a single pipeline for the whole of Notify. The versions of the different apps which make up a deployment are dictated by the resources that are input to the pack-bag job.

• You should pin the versions of each resource while you are developing a feature, as this ensures that changes to the latest versions of other apps do not interfere with the effects of changes you are working on.

Each resource input to the pack-bag job is sourced from an AWS Parameter Store (SSM) variable, specifically in the notify-deploy AWS account in the London region (eu-west-2). NB other environment specific AWS accounts all reside in the Ireland region (eu-west-1).

To restore all SSM parameters used by the concourse deploy pipeline to their default values:

1. In your Terminal, navigate to your notifications-aws/concourse/deploy-notifydirectory.
2. Use the following make target, where <env> is formatted like "dev-c":

gds aws notify-deploy-devsecrets -- make <env> set-pipeline-vars

There may be some errors relating to missing items in the password store. This is normal, as these are creds used by other builds.

The way that you set custom pack-bag input resources in dev environments is different depending on whether the resource is a git-repo concourse resource (e.g. ) or a docker-image concourse resource (e.g. ).

If you want to test changes to configuration repos, like notifications-aws, you will need to make changes to git-repo resources, as follows:

• Git-repo resources such as notifications-aws can be modified to use code from a specific branch by altering the value of the corresponding <resource>_branch variable in SSM Parameter Store. Be aware that resources in concourse appear with dashes "-", but their corresponding resource paths in SSM use underscores "_".

If you want to test changes to apps, like notifications-api or notifications-admin, you will need to make changes to docker-image resources, as follows:

• Docker-image resources can be modified to use specific demo images by altering two SSM parameters:
  • The <resource>_tag_regex parameter
  • The <resource>_repo_name parameter

Note that docker images are built using the existing pipelines in the notify concourse team, usually in the pull-requests team. The tag corresponding to the image you wish to use is defined there and the name of the repo that demo image gets pushed to is also there.

Log into the deploy-notify console in the London region:

gds aws notify-deploy-devsecrets -l

Navigate to Parameter Store (ensuring the London eu-west-2 region is selected):

Locate the SSM parameter corresponding to the resource you wish to modify to a different source branch or image tag.

• For the branch of a git-repo resource you need to:

  1. locate the _branch_name parameter - that's a parameter with branch in the name, for example /concourse/concourse/pipelines/dev-b/deploy-notify/notifications_deployment_bag_repo_branch.
  2. Click that resource, Edit it, and enter the branch name in the "Value" textbox.

• For a docker image resource you need to locate and edit two parameters:

  1. The _tag_regex parameter: find it, edit, and in Value textbox, enter a regex matching the PR you're using to build your dev PR image. The default regex should be ^\d+-\d+-\d+.*, you will modify to a regex to something like ^pr-4321-.*).
  2. The _repo_name parameter: find it, edit, and in Value textbox, set it to "<app-name>-demo".

You can ignore this error, the devsecrets account has limited permissions intentionally:

Return to concourse and select the team for your dev environment. In the pack-bag tab you can now click into that resource and verify that the latest hash of the commit (if a git-repo resource) or hash of the target image in ECR is correct. It may take a few minutes for concourse to detect changes to it’s SSM source variables and retrieve the correct resources:

Once you have set up the required inputs to your pack-bag and pinned the versions of every resource that you are not working on, you can start the pack-bag job to begin a deployment of the dev environment.

If you find that the deployment seems to hang at the start-deploy job, specifically on the step acquiring the pipeline-lock, you may need to forcibly unlock the pipeline using the force-unlock-pipeline job under the operator tab (⚠️take care that no jobs are running that have legitimately taken the lock).

As the deployment progresses through the jobs of the deploy pipeline the same deployment-bag is fetched by each job, assuring that the same versions of each app are played together. You must allow the jobs to run in sequence before you can assume that the latest deployment is available (e.g. don't force the tests job to run before the deploy job has completed).

If a job fails for transient reasons you can rerun it by clicking into the job and clicking "trigger a new build". Because of the way the deployment-bag is consumed, it will run idempotently as long as the same deployment-bag is still available (if you have modified variables in SSM since deployment you may find old deployment bags are no longer available).

During development you are advised to pin all inputs to the pack-bag except the ones you are actively working on. The reason to do this is so that other changes happening in the background do not impact your work while developing.

There is a script to pin and unpin all resources for convenience. To use it login to concourse using the fly CLI tool where <env> is like "dev-c":

fly -t notify-<env> login

Then go to notifications-aws/concourse/deploy-notify and run:

• make <env> unpin-bag-resources to unpin all resources

• make <env> pin-bag-resources to pin all resources.

Well done, you have now set up your dev env 🎉 . You can now access your custom Notify app at:

https://www.<env>.notify-<env>.space  # substitute <env> with the name of your environment, for example dev-b

Important note. The instructions in this sub section do not apply to dev-livemail. In dev-livemail SES is in production mode and can send emails to any address.

The Simple Email Service (SES) in the dev environments (with the exception of dev-livemail) is running in sandbox mode. When SES is in the sandbox mode, you can only send email to verified identities. A verified identity is an email addresses that you've proven that you own. Additionally, when SES is in sandbox mode, there are limits to the volume of email you can send each day, and to the number of messages you can send each second.

Therefore to sign up for a Notify account or send email notifications to yourself via Notify in a dev environment, you must make sure your email address is configured as a verified identity in SES.

SES sandbox identities are not managed via terraform, and should be manually added to each dev environment that you are using. To do this, log in to the dev environment with full admin privileges with the following command:

gds aws notify-<env>-admin -l

From the AWS console, choose Amazon Simple Email Service (SES) using the services search menu. Then click Identities under the Configuration menu on the left hand menu. Here you will see a list of identities set up in SES.

Use the "Create Identify" button to add a new email address (not domain) to the list that SES is allowed to send to. SES will send a verification message with a magic link to the email address that you configured to verify you are able to access the address. Once an identity is verified, the dev environment will be able to send emails to that address.

The pipeline itself is defined as a jinja2 template deploy-notify.yml.j2 at notifications-aws/concourse/deploy-notify.

The template is built using a script called render-pipeline.sh.

You can make amendments to how the pipeline is constructed by setting Jinja2 variables then running the set-pipeline make target. You can see what Jinja2 variables are available by looking in the render-pipeline.sh script.

To do this firstly login to concourse using the fly CLI tool where <env> is like "dev-c".

fly -t notify-<env> login

Then set a Jinja variable before running the make target from the same directory (notifications-aws/concourse/deploy-notify):

JINJA_BOOL_include_continuity_tests=1 make <env> set-pipeline

Viewing in concourse you will now see the continuity test job appears as part of the pipeline:

"Shutting down" currently means destroying. This can be done by going to the destroy tab of the pipeline and triggering a new "start-destroy" job. (Anything clickops'd will be lost). Other jobs need to be finished before running destroy (it shares the pipeline lock).

After the destroy is complete go back to the pack bag and unpin the components (so the environment is ready for the next person to configure).

Don't forget to edit the slack channel topic on #govuk-notify-infrastructure-team to show that the dev environment that has been destroyed is now available for reuse.