Concourse Upgrade - alphagov/notifications-manuals GitHub Wiki

This page will mention considerations for when Concourse is upgraded. In most cases you can ignore the content here as Concourse updates - even major ones - don't tend to be breaking. This page assumes you are familiar with:

• notifications-concourse, notifications-concourse-deployment.
• the difference between concourse-web and concourse-workers.

Previous upgrade ticket contains more details if required.

NOTE: These instructions assume you are doing a minor upgrade. If it is a major upgrade please read through the considerations section before starting.

1. Spin up the Concourse staging environment, intructions here.

2. Create a new branch of notifications-concourse and set this branch as the reference in concourse-deployer-staging.vars

module_repo_uri: [email protected]:alphagov/notifications-concourse.git
module_repo_branch: <new branch name here>

3. Update the Concourse version here by updating semver_restraint's value

- name: concourse-release
  type: github-release
  check_every: 30m
  source:
    owner: concourse
    repository: concourse
    semver_constraint: <new-value-here> # No prefixing 'v' required

4. Deploy the changes using these instructions.

5. Check the new version has been deployed for both concourse-web and concourse-worker (the version label in the bottom right of Concourse is not enough!) Log into the notify-deploy-staging

gds aws notify-deploy-staging-admin -l 

EC2 > Instances > For both concourse-concourse-web and concourse-main-concourse-worker SSM onto them > Check the concourse version being run:

/usr/local/concourse/bin/concourse <web | worker>  --version
8.0.1 

You must do this for all web and worker instances.

If the version is incorrect try the following:

• Ensure the deploy pipeline has completed
• Manually kill off each instance and wait for it to come up. Note, if you kill off all web workers at once you cannot access the UI, so I'd recommend one at a time.

6. Check that the packbag resource works. Instructions.

After the scale-out job you may get issues with concourse-web e.g. UI not working. This is because the old instances have not yet scaled in and there is some compatibility issue between workers using the old version and the new version. In some cases the scale-in action will be triggered. You can check this in the relevant autoscaling groups tab in the AWS console. It is also safe to manually terminate the old workers.

Sometimes after upgrading the input requirements for a job may never be satisfied, desipite looking healthy. You should start the pipeline again from the beginning.

You must roll all workers after upgrading the concourse version. Otherwise they will roll after hours and any issues may cause an on-call alert.

In-use dev envs may require a rebase against notifications-aws if the pipeline config has changed as part of the new Concourse version.

Our main pipeline does not use the concourse-bag-resource, but it must be tested for compatibility before we roll out any upgrades. This is a custom resource written by Rob.

These PRs:

1. https://github.com/alphagov/notifications-concourse-deployment/pull/117/changes
2. https://github.com/alphagov/notifications-concourse/pull/45/changes 3.https://github.com/alphagov/notifications-concourse/pull/44/changes

contains the config required to test the deployment bag in concourse-staging. There are also some manual changes which need to happen. This setup will test the get actions of this resource, but not the put ones. Historically, if the get actions pass it has been proof enough that the deployment bag resource is compatible. Getting the put actions working is not covered here, but would be a more involved process, giving the Concourse team the relevant key.

1. in notify-deploy-staging manually create ECR repo, concourse-bag-resource.

2. Log into ECR in notify-deploy

gds aws notify-deploy -- aws ecr get-login-password --region eu-west-1 \
| docker login \
--username AWS \
--password-stdin 975210404583.dkr.ecr.eu-west-2.amazonaws.com

3. Pull the image down docker pull 975210404583.dkr.ecr.eu-west-2.amazonaws.com/concourse-bag-resource:latest

4. Log into notify-deploy-staging's ECR

gds aws notify-deploy-staging-admin -- aws ecr get-login-password --region eu-west-1 \
| docker login \
--username AWS \
--password-stdin 390844751771.dkr.ecr.eu-west-1.amazonaws.com

5. Retag the old image in the format docker tag <aws account id you pulled from >.dkr.ecr.eu-west-1.amazonaws.com/concourse-bag-resource:latest <aws account if you want to push to>.dkr.ecr.eu-west-1.amazonaws.com/concourse-bag-resource:latest e.g.

docker tag 975210404583.dkr.ecr.eu-west-1.amazonaws.com/concourse-bag-resource:latest 390844751771.dkr.ecr.eu-west-1.amazonaws.com/concourse-bag-resource:latest

6. Push the image to the notify-deploy-staging account

gds aws notify-deploy-staging-admin -- docker push 390844751771.dkr.ecr.eu-west-1.amazonaws.com/notifications-api:2026-03-05-11.29.40-c405671ce8d3ccfbcb59647810cb5f07215a3088

7. Check your concourse-deployment-bag image exists in the eu-west-1 region.

1. The main team is accessed with an AWS user concourse-<staging>-main-concourse-worker-static user. This user must have access to the concourse-bag-resource a policy like this will do

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": "ecr:GetAuthorizationToken",
            "Effect": "Allow",
            "Resource": "*",
            "Sid": ""
        }
    ]
}

2. Check the permissions on the concourse-bag-repo allow the concourse-<staging>-main-concourse-worker-static principal to access the required resources e.g.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Access_deployment_bag",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::390844751771:user/concourse-staging-main-concourse-worker-static"
      },
      "Action": [
        "ecr:GetDownloadUrlForLayer",
        "ecr:GetLifecyclePolicy",
        "ecr:GetLifecyclePolicyPreview",
        "ecr:GetRepositoryPolicy",
        "ecr:DescribeImageScanFindings",
        "ecr:DescribeImages",
        "ecr:DescribeRepositories",
        "ecr:ListImages",
        "ecr:ListTagsForResource",
        "ecr:BatchGetImage",
        "ecr:BatchCheckLayerAvailability"
      ]
    }
  ]
}

The Concourse DB can be rolled out via the same Terraform mechanism we have in notifications-aws. Instructions for how to do that can be found here.

If you decide to create this database and switch over to it locally you must commit this change and update the concourse-staging branches for notifications-concourse and notifications-concourse-deployment or when you next run the pipeline you will destroy the database. If you're paranoid you can pause the deploy job.

If your Concourse upgrade contains schema changes then you must take a snapshot of the DB using the old schema beforehand. You should clean these up at some point after the upgrade is deployed.

Check the currently deployed systemd unit for concourse worker / web

cat /etc/systemd/system/concourse-worker.service

Check it's running

systemd status | grep concourse 

Check logs if there's weirdness

journalctl -u <unit name>

Try this same page in a private tab to see if this is the issue - it has been before!