Post deployment test runbook - guardian/support-frontend GitHub Wiki

Don't panic!

This document is here to help you figure out if the tests have failed because of a problem with the tests or a problem with the website.

If there's a problem with the website, you'll want to reproduce the issue and decide if we can rollback or if we need to fix forward.

If there's a problem with the tests, you can take a breath and find out if the issue is permanent or transient, and if it happened with the automated run, the build or the actual tests.

1. Let others know!

You might have gotten a message on your PR or noticed the message on google chat. Reply to the automated message on google chat to let others know you're looking into it!

2. Check the cron or smoke test run workflow

The Playwright Smoke Tests workflow runs on every pull_request to main that also has the Seen-on-PROD label attached to it (from prout). The action runs a suite of Playwright tests against the live site. Similarly the Playwright Cron Tests workflow runs hourly during working hours. It runs similar tests but using PayPal as the payment method, which we need to rate limit otherwise there's a risk of being blocked by PayPal.

If there's been a failure, the first place to look is as the output of this command in the failing build's corresponding workflow run. To see the failing build open the relevant workflow run. Click into the details and check the build log to see what failed. You might need to scroll down to the bottom.

You have the option to re-run the failed job.

3. Check the Playwright build artifact

At the end of each Playwright workflow run an artifact playwright-report is uploaded to GitHub. In the case of failures this artifact can be downloaded and inspected locally to find out more about the failure. Inside the ZIP file there is an index.html which can be opened in a browser.