Deploying a demo app image to preview before merging - alphagov/notifications-manuals GitHub Wiki

This is a quick way to get extra confidence about a change before merging it.

This process is currently a little awkward on ECS, and it will be until we have properly independent environments for development.

Besides running the relevant tests, the concourse job performing pre-merge CI for a pull-request will also build and upload a "demo" image built for the docker "production" target. You need to find the sha256 of this image:

It is pushed to a docker repo suffixed with -demo, under the demo-production tag (using unique tag names would give us an additional garbage collection problem to solve, and referencing images by sha256 is more precise anyway - they should stick around long enough to be used for testing).

Before you start this, it's probably a good idea to warn people in an appropriate slack channel and pause the concourse job for that app's deployment to preview, to avoid having your deployment reverted before you're ready.

In the aws console, find the ECS service you would like to replace the app image for:

Under its Tasks tab:

Select the Task Definition revision used by the current ("Primary") deployment:

We're going to edit the image reference of the app's container definition, for which you can see the existing value under the JSON tab:

From the Create new revision drop-down select Create new revision with JSON:

Append -demo to the repo name and change the tag-specifying section to @sha256:<sha256-copied-from-earlier>:

Now this needs to be deployed - from the Deploy dropdown choose to update a service:

Choose the respective service:

The rest of the settings for the service should carry over their correct values, but you might like to choose to enable the "circuit breaker" for this deployment, which will detect whether the new apps have failed to come up properly and stop the deployment (otherwise a failing deployment will sit trying to restart the new apps for a long time). Note however that using this feature may require you to run both the notify-infra pipeline job and the app pipeline job to restore the deployment back to its normal state (discussed later).

Hit Update:

Now the fun begins watching the new Primary deployment hopefully taking over the service and then the previous Active one being shut down. If the deployment fails, you can try deciphering the ECS event listing, finding the task id(s) that failed to start properly and look at their log output - either from the ECS task interface or directly in CloudWatch.

Once you're done, unpause the preview deployment concourse job and trigger a new run of it - it should re-deploy a previous task definition and bring everything back as it should be.

If you made changes to the ECS service other than the image, you may find the app pipeline refusing to deploy over this apparent configuration drift.

In this case you should probably be able to re-run the respective notify-infra job preview and it will restore that configuration. At which point you should be able to run the app pipeline successfully.