Notifications aws local Terraform - alphagov/notifications-manuals GitHub Wiki

Note

Applying Terraform locally is discouraged and should Terraform changes should be done via pipeline in most cases. Where this is not possible local changes should only be applied to dev environments. Changes to staging or production must happen via a pipeline after PR approval.

This page is intended for engineers who want to see Terraform changes locally against the Notify applications or infra. For changing bootstrap infra see the account-wide Terraform page.

This page assumes knowledge of:

  • Terraform basics (inc state and interpreting Terraform plans)
  • Makefile basics
  • Deployment bag resource
  • How to change resources deployed on a dev env.

Setup

To apply changes locally you must have notifications-deployment-bag-dev and notifications-deployment-bag-live cloned in the same dir that notifications_aws is in i.e.

.
├── notifications-aws
├── notifications-deployment-bag-dev
├── notifications-deployment-bag-live
├── # Other repos

In notifications-deployment-bag-dev (all dev envs) or notifications-deployment-bag-live (staging and production), the branch must be on the branch corresponding to the env you want to apply changes for i.e. for local changes to dev-c , notifications-deployment-bag-dev must be checked out to dev-c to ensure your Terraform commands are run against the state for the environment you want them to run against. Further reading on packbag resource.


Warning

These repos must have the latest commits in your local copy before running any Terraform commands locally. The Terraform plan is run against the notification-aws input to the deployment back, not the Terraform files in your local repo.


What happens here

For simplicity this is written as if an app (docker resource) has been changed. It is assumed this docker resource is unpinned.

On a PR when you make a commit to an app repo, that app is built into a 'demo' image which has a digest and is tagged with the PR number.

This new demo image causes a new docker resource to be 'found' which then updates the contents of the pack bag. The pack bag update then makes a commit to the deployment-bag-dev resource containing the image digest and tag of the new resource on the corresponding branch to the env you're working on i.e. if you're making changes to dev-c, then this commit will be to the dev-c branch.



Applying local changes

Note

Applying Terraform locally is discouraged and should Terraform changes should be done via pipeline in most cases. Where this is not possible local changes should only be applied to dev environments. Changes to staging or production must happen via a pipeline after PR approval.

Changes applied locally which are not committed to the notifications-aws repo will be overwritten on the next resource run.


We can now apply any changes locally.

cd /path/to/notifications-aws/terraform/notify-infra | notify-grafana

In the dir tfvars open the vars file for the env you're working on e.g. dev-c.tfvars and make the changes you wish to there e.g.

antivirus_worker = {
  cpu = 1024
  autoscaling = {          
    min_capacity         = 3
    max_capacity         = 3

becomes

    min_capacity         = 5
    max_capacity         = 5

Then run this to generate your Terraform plan which will show you which changes will be applied

gds aws notify-dev-<a-f>-admin -- make dev-<-a-f> plan

# module.antivirus_worker.aws_appautoscaling_target.scale_target[0] will be updated in-place
  ~ resource "aws_appautoscaling_target" "scale_target" {
        id                 = "service/notify/antivirus-worker"
      ~ max_capacity       = 3 -> 5
      ~ min_capacity       = 3 -> 5

If you are happy with the changes then run

gds aws notify-dev-<a-f>-admin -- make dev-<a-f> apply

Checking config drift

You can also use a local terraform plan to check any config drift which may have occurred. Here you can see that the plan wants to change the min and max capacity for the antivirus_working from 6 to 3. 3 is declared in the tfenvs file, indicating config drift.

# dev-c.tfvars
antivirus_worker = {
  cpu = 1024
  autoscaling = {
    min_capacity         = 3
    max_capacity         = 3

Changes to notifications-aws on a branch

If you are testing changes to the notifications-aws repo and want to make local Terraform changes there is greater scope for confusion. Each time you re run your pipeline any local changes will be overridden by the contents of notifications-aws in the pack bag.

Troubleshooting

Deployment bag location does not exist

You have not cloned the deployment bag repo into the correct dir.

suspended_state

When a dev env is shut down overnight it is put in a 'suspended' state and will remain so until it is 'woken' up again. How to wake it.

⚠️ **GitHub.com Fallback** ⚠️