Dependencies - alphagov/notifications-manuals GitHub Wiki

Dependency management

We want to review dependabot vulnerability alert severity on a weekly basis. We know this may take some time initially, but as we reach a point where we have caught up with releases the pressure should decrease. This process will evolve and change over time.

From Q1 a software engineer from both Dove and Pigeon will spend one day a fortnight working on the following dependency tasks:

Spend around 60% of the day doing the following for each repo in turn. Do as many repos as time allows, keeping notes in the next repo to update dependencies card. While working on dependency updates for the repo:

  • Follow the instructions on the team wiki to unfreeze / update / freeze non pinned dependencies on the chosen repo. Test and deploy.
  • For template-preview only freeze / update / refreeze non PDF dependencies.
  • Check logit to check for any issues introduced by a dependency update using the following commands;
    • status: [500 TO 599] OR levelname: ERROR
    • *-critical
  • Check any remaining dependabot alerts not fixed by the freeze/unfreeze process (likely those for pinned dependencies, or where a fix for a dependency is still withing the 7 day cooldown period). Investigate and triage each remaining dependabot alert, keeping notes in the relevant trello card on the vulnerability tracking board (created by the dependabot to trello script). Move the card to:
    • Fix - dependency issues that could affect Notify and should be fixed quickly
    • Non urgent fix - dependency issues that need to be fixed but do not pose a significant security threat to Notify.
    • Acknowledge - dependency updates that we have decided do not need to be fixed (with a detailed explanation of why)
    • Investigate - dependencies which we do not believe need to be fixed quickly, but need further analysis from a colleague.
  • Investigate and resolve items from the Fix column in the vulnerability tracking board, keeping notes on the card. Test and deploy where possible. Flag any urgent cards to the team that you might need help with, or where deploying an updated pinned dependency might break something. Decide what to do about vulnerabilities from unpinned dependencies where the fix is applied to an upgrade still within the 7 day cooldown window.

Then for the remaining 40% of the day

  • Check the Investigate and Non urgent fix columns in the vulnerability tracking board for the repos you have just processed. Prioritise which cards to work on, and then try to resolve these issues by upgrading the pinned dependencies. Keep notes on the card (for the next person).
  • Ask Ian to run the dependabot to trello tool again, to automatically archive all of the card for which fixes have been applied.
  • Run the make show-outdated-requirements tool on the repo, and update any pinned dependencies where it is sensible to do so.

Q2 onwards: Ishwar and Ian review any long open vulnerabilities once a month and decide if those can be closed or require any fix.

Python

We upgrade to the latest bugfix version of Python once a year.

Python packages

Where dependencies are specified

Each app has:

  • requirements.in – application dependencies
  • requirements_for_test.in – additional dependencies for local development and running tests
  • requirements_for_test_common.in – common dependencies for local development and running tests (automatically copied from notifications-utils)

These files are used to generate:

  • requirements.txt - pinned versions of all application dependencies (installed when building apps to run on ECS)
  • requirements_for_test.txt - pinned versions of all application, development and test dependencies (installed locally and on CI)

How to add a new dependency

  1. Add the dependency to the appropriate *.in file
  2. Run make freeze-requirements to regenerate the *.txt files

For Dockerised apps, you’ll need to run ./scripts/run_with_docker.sh make freeze-requirements.

How to specify dependencies

In requirements.in

Use the compatible release operator with a major and minor version:

example~=1.2

This means greater than or equal to 1.2.0, but less than the next major version (2.0.0)

In requirements_for_test.in

Do not specify a version:

example

If you need a more exact version

For the latest patch version, use

example~=1.2.3  # A comment explaining why this is necessary

For an exact version, use

example==1.2.3  # A comment explaining why this is necessary

How to upgrade dependencies

To upgrade all dependencies to the latest constrained version

  1. Run make refreeze-requirements
  2. Raise a pull request

By default make refreeze-requirements will not fetch dependencies less than 7 days old. If it’s downgraded any existing dependencies you should manually revert those lines.

Upgrading a dependency’s constraints

  1. Run make show-outdated-dependencies
  2. If a dependency is flagged with ‘upgrade needed’ then edit requirements.in
  3. Run make refreeze-requirements
  4. Raise a pull request

If updating a major version causes the tests to fail then reading the dependency’s changelog can help understand why.

Consider consolidating multiple upgrades into a single pull request if they don’t require code changes.

Consider if there are any dependencies whose constraints can be loosened (for example from ~=1.2.3 to ~=1.2).

If updating a dependency breaks something

Have a strong bias towards fixing the broken thing, rather than constraining the dependency.

If you need to constrain a dependency to anything other than the latest major version, you must:

  • add a comment explaining what is blocking it from being updated
  • put the work to fix it into a backlog

If you need a very fresh version of a dependency (released in the last 7 days)

  1. Make a file called uv-overrides.toml in the root of the repo, following this example:

    exclude-newer-package = {
    flask = "1 day"  # Can be removed after dd/mm/yyyy
}

    This will let uv install versions of flask that are more than 1 day old, rather than the default of 7 days.

  2. Run make freeze-requirements

  3. (if needed) edit requirements.in or requirements_for_test.in with the version you want

  4. Run make refreeze-requirements

NPM packages

We use specific version numbers, not semantic version ranges so, when installing new versions of packages, use the --save-exact flag.

Security updates

We have a weekly job that audits Production packages for security vulnerabilities:

  • Check the output in Concourse to see which packages have vulnerabilities.
  • Make a PR to update affected packages to their patched versions.

If the packages are listed in package.json under dependencies, update the versions there and then run npm install to update the lockfile. For sub-dependencies, there's unfortunately no easy way to tell NPM to "update just this package and anything that conflicts with it":

  • Try updating just the affected packages in package-lock.json and then run npm ci to see if there are any conflicts.
  • If you see conflicts with other packages, it's usually safe to just run a dumb npm install, which will update everything.

General updates

Currently we do this as-and-when. We should prioritise dependencies with security vulnerabilities.

Docker images

Security updates

We have a weekly job that scans Production Docker images for vulnerable packages, in this repo

These images are stored in the notify-deploy account.

The resources were created to allow scan of images on the notify-deploy account.

An event bridge is scheduled to run once per week triggering the lambda function.

The lambda function scans the ECR repository to list the repositories and it also finds the very latest image version pushed by date.

After that if starts the scan on the images and if any CRITICAL vulnerability is found, it aggregates the results into a findings.txt file that will be sent to a SNS group at the end in the form of an email with the subject: ECR SCAN-Critical Vulnerability Report