Concourse Database Upgrade - alphagov/notifications-manuals GitHub Wiki

This document will talk about some considerations when upgrading Concourse's database version, not the Grafana database. It assumes you have an understanding of Concourse and the notifications-concourse and notifications-concourse-deployment repos.

There are cron jobs you may wish to not interrupt with the upgrade (e.g. Antivirus rebuild at 1100.)

You should test the upgrade on Concourse staging first. There may be changes in the RDS parameter group which would cause incompatibilities with your current launch template for example, in Postgres 13 rds.force_ssl is false, but in Postgres 15 is true. Reverting DB upgrades requires restoring from a snapshot, an involved process.

A snapshot will be taken automatically before upgrading the database and the whole upgrade process (including backups) will take about 8 minutes.

If changes to the launch template (LT) are required then you may want to see if the changes are compatible with the current version of the database. If they are then you may want to merge updates to the launch template and roll them about before merging in the database upgrade.

If you merge both DB and LT in one then you may need to:

1. Check the new launch template version exists
2. Log in as admin to notify-deploy and terminate all the concourse-concourse-web instances.
3. Wait for the instances using the new LT to come up. It may take a few minutes for Concourse to come back up.

This method caused about 15 mins of downtime. This is the deploy job for a DB upgrade. For this update the process from kicking off the pipeline, to the pipeline finishing, to Concourse being usable again was about 1 hour.