Concourse Recovery Plan - alphagov/notifications-manuals GitHub Wiki

This page should not be considered usable until the WIP marker above has been reviewed. This content is part of an on-going ticket and is incomplete and unreviewed.

This page will cover the 3 likeliest scenarios that Concourse would need to be recovered:

1. AWS region is down
2. Concourse app itself is compromised
3. Concourse update destroys our settings and we cannot roll back

Warning: This document only covers deploying Concourse in the case of an eu-west-2 outage, not other Notify infra, even though changes are required in notifications-aws too.

This pages assumes you know about:

• Signing into an AWS account with the admin role.
• notifications-concourse-deployment and notifications-concourse; how to use them and their purpose. A primer can be found here (1, 2).

These methods are untested as was agreed to keep the scope of this ticket small.

Warning: If us-east-1 is down then there is nothing we can do if changes are required for various AWS services which use us-east-1's control plane. This is a known and accepted risk.

Concourse instances run in eu-west-2 by default and the buckets containing their state does too. Most of Notify's infra runs in this eu-west-1. It is assumed you already know the new region you will deploy into. The examples below assume you have chosen to deploy into eu-west-1.

The first step is to manually create an S3 bucket to store terraform state in the notify-deploy account. This should be called notify-deploy-tfstate-<new region-name>-<6 random characters>.

Then update this value in both notifications-concourse-deployment and notifications-concourse, where the current bucket name, notify-deploy-tfstate, is set. In notifications-concourse-deployment you must also update the region to your new one e.g. eu-west-1.

backend "s3" {
    bucket = "notify-deploy-tfstate"
    # should be <deployment_name>.tfstate
    key    = "concourse.tfstate"
    region = "eu-west-2".  <- This becomes "eu-west-1"
  }

You must update the region where it is set to eu-west-2 in all locations.

Apart from locations where the region is a comment e.g. pipelines/concourse-deployer.vars.yml line 38

You must merge these changes first.

You must update the region where it is set to eu-west-2 in all locations. You must not update the region for resource "aws_ecr_replication_configuration" "cross_region_replication".

You must run scripts/upload-credentials/upload-credentials.sh so the expected credentials exist in the concourse env's (i.e. concourse or staging-concourse) SSM parameter store in the new region.

You must merge these changes second.

Once merged you should then manually create infra from the notifications-concourse-deployment repo with a second pair of eyes by running Terraform locally.

Security issues can be found here.

Update to the version of Concourse where this issue is fixed. The the security page contains the issue and the Concourse version it has been fixed in.

This process is not guaranteed to have zero-downtime. You must also check this page in case any new specified upgrade paths are dictated.

1. Post in #govuk-notify-tech that you are upgrading Concourse. People should not merge anything until this is complete and Concourse jobs may fail while the downgrade is happening. People must not trigger a new build in the main team's deploy pipeline, update job. You must pause this job.
2. Set the new version here and merge it in.
3. Unpause the deploy pipeline's update job. Then manually trigger the main Concourse team (pipeline) and you should see the new Concourse version be deployed as part of the deploy / concourse-release job.
4. Once completed update in #govuk-notify-tech that people can merge and normal pipeline functionality has resumed.

There is no official downgrade path that applies to our use case, but Concourse's suggested method has been adapted here for our set up. It is assumed you know the Concourse version you want to roll back to before starting.

We are restoring from a database snapshot, as this is a simpler downgrade path and an minimal amount of history will be lost. This method will have downtime.

You must have a database snapshot from before the migration. You cannot use a database which has been upgraded with a Concourse version that has been downgraded. You must use the most recent snapshot feasible.

1. Post in #govuk-notify-tech that you are downgrading Concourse. People should not merge anything until this is complete and Concourse jobs will fail while the downgrade is happening. People must not trigger a new build in the main team's deploy pipeline, update job. You must pause this job.
   
   
2. In the notify-deploy account under EC2, alter the concourse-concourse-web autoscaling group (it may be on the second page). Change Desired capacity to 0, Max desired capacity to 0 > click Update.

3. Wait for the number of instances called concourse-concourse-web to be 0. Force terminating instances will break running Concourse tasks.
4. Sign into notify-deploy as admin and follow the Console instructions for restoring from a DB snapshot here. Note this will be a snapshot named, rds:concourse-concourse-*, not rds:concourse-concourse-grafana-*.
5. Once restored, update the version of Concourse you want to downgrade to in notifications-concourse.
6. Merge this in, unpause the update job.
7. Let the pipeline deploy the new version of Concourse.
8. Once completed update in #govuk-notify-tech that people can merge and normal pipeline functionality has resumed.
   
   

We are not restoring an older database snapshot because we would lose too much history.

1. Post in #govuk-notify-tech that you are downgrading Concourse. People should not merge anything until this is complete and Concourse jobs may fail while the downgrade is happening. People must not trigger a new build in the main team's deploy pipeline. You must pause this job.

2. Sign into notify-deploy as admin and go to EC2 > Autoscaling Groups (asg)

3. Select the concourse-concourse-web asg, click Edit

4. Change Desired capacity to 1, Max desired capacity to 1 > click Update.
   
   

5. Wait for the number of instances called concourse-concourse-web to be 1. Force terminating instances will break running Concourse tasks.

6. Session manager onto the remaining concourse-concourse-web instance.

7. Follow the downgrade instructions here

8. You must then update the version of Concourse you have downgraded to in step 7 in notifications-concourse

9. Unpause the deploy pipeline and merge in these changes and by going to Concourse > main team > deploy pipeline > update job > trigger a new build. The rollback version will then be deployed.

10. Once completed update in #govuk-notify-tech that people can merge and normal pipeline functionality has resumed.

Follow the process in Rolling back within 0-2 days of upgrading