Rollback Procedures - bcgov/eagle-dev-guides GitHub Wiki

Rollback Procedures

This document describes how to rollback deployments when issues are discovered in test or production environments.

Rollback Methods

There are two ways to rollback:

Method Use Case Complexity
Version Redeploy Rollback to any previous release Simple - Use existing workflow
Helm Rollback Quick rollback to previous Helm revision Quick - CLI command

Method 1: Version Redeploy (Recommended)

The deploy-to-test and deploy-to-prod workflows support deploying any existing version, making rollbacks simple.

Rollback Test Environment

  1. Identify the version to rollback to:

    # List available versions (Git tags)
git tag --list 'v*' --sort=-version:refname | head -10

# Or check GitHub releases
gh release list --limit 10

  2. Run Deploy to Test workflow:

    • Go to Actions → Deploy to Test
    • Click Run workflow
    • Enter the previous version (e.g., v1.1.0)
    • Click Run workflow

  3. Workflow will:

    • Detect tag already exists (rollback mode)
    • Verify image tag exists in registry
    • Tag that version as test
    • Restart deployment
flowchart LR
    A[Input: v1.1.0] --> B{Tag exists?}
    B -->|Yes| C[Verify image exists]
    C --> D[Tag v1.1.0 → test]
    D --> E[Rollout restart]
    E --> F[✅ Rolled back]

Rollback Production Environment

  1. Identify the version to rollback to:

    # List versions that have been deployed
git tag --list 'v*' --sort=-version:refname | head -10

  2. Run Deploy to Prod workflow:

    • Go to Actions → Deploy to Prod
    • Click Run workflow
    • Enter the previous version (e.g., v1.1.0)
    • Click Run workflow

  3. Workflow will:

    • Verify Git tag exists
    • Verify image tag exists
    • Tag that version as prod
    • Restart deployment

⚠️ Note: Only versions that have been previously deployed to test will have image tags. You cannot skip directly to prod with a version that was never deployed to test.

Method 2: Helm Rollback (Quick)

For quick rollbacks to the immediately previous deployment, use Helm directly.

View Revision History

# List deployment history
helm history eagle-public -n 6cdc9e-test

Output:

REVISION  UPDATED                   STATUS      CHART              APP VERSION  DESCRIPTION
1         Fri Feb  6 15:30:00 2026  superseded  eagle-public-1.0.0 1.0.0       Install complete
2         Fri Feb  6 18:00:00 2026  superseded  eagle-public-1.0.0 1.0.0       Upgrade complete
3         Fri Feb  6 20:00:00 2026  deployed    eagle-public-1.0.0 1.0.0       Upgrade complete

Rollback to Previous Revision

# Rollback to immediately previous revision
helm rollback eagle-public -n 6cdc9e-test

# Rollback to specific revision
helm rollback eagle-public 1 -n 6cdc9e-test

Verify Rollback

# Check current revision
helm status eagle-public -n 6cdc9e-test

# Check pods are running
oc get pods -n 6cdc9e-test -l app.kubernetes.io/name=eagle-public

# Check pod image
oc get pods -n 6cdc9e-test -l app.kubernetes.io/name=eagle-public -o jsonpath='{.items[0].spec.containers[0].image}'

Emergency Rollback Checklist

🚨 Production Issue Detected

  • Assess severity - Is the site down? Data corruption? UI bug?
  • Communicate - Notify team in #falcon-general
  • Identify last known good version 
    gh release list --limit 5
  • Execute rollback
    • GitHub Actions → Deploy to Prod → Enter previous version
  • Verify rollback successful
  • Document incident - Note what happened and version rolled back to
  • Investigate root cause - Review the problematic version

Finding Previous Versions

GitHub Releases

# List recent releases
gh release list --limit 10

# View release notes for a version
gh release view v1.1.0

Git Tags

# List all version tags (newest first)
git tag --list 'v*' --sort=-version:refname

# See what commit a tag points to
git log -1 v1.1.0 --oneline

# Compare two versions
git log v1.1.0..v1.2.0 --oneline

Image Registry

# List available image tags
oc get imagestreamtag -n 6cdc9e-tools | grep eagle-public

# Check when an image was pushed
oc describe imagestreamtag eagle-public:v1.1.0 -n 6cdc9e-tools | grep Created

Rollback Scenarios

Scenario 1: Bad Deploy to Test

Situation: Deployed v1.2.0 to test, found critical bug.

Solution:

  1. Run Deploy to Test with v1.1.0
  2. Fix the bug in code
  3. Merge fix to develop (creates new ci-latest)
  4. Deploy v1.2.1 to test

Scenario 2: Bad Deploy to Production

Situation: Deployed v1.2.0 to production, users reporting errors.

Solution:

  1. Immediately run Deploy to Prod with v1.1.0
  2. Verify production is stable
  3. Investigate and fix v1.2.0 issues
  4. Test fix thoroughly
  5. Deploy v1.2.1 when ready

Scenario 3: Need to Rollback Multiple Versions

Situation: Issues found in v1.2.0, v1.1.0, need to go back to v1.0.0.

Solution:

  1. Verify v1.0.0 image tag exists: 
    oc get imagestreamtag eagle-public:v1.0.0 -n 6cdc9e-tools
  2. If exists, deploy v1.0.0 using workflow
  3. If not exists, you'll need to rebuild from that Git tag

Scenario 4: Rollback Dev Environment

Situation: Dev environment is broken, need to restore.

Solution: Dev uses mutable dev and ci-latest tags, so you cannot easily rollback to a previous dev build. Options:

  1. Deploy a known version to dev temporarily:

    helm upgrade --install eagle-public ./helm/eagle-public \
  --namespace 6cdc9e-dev \
  --values ./helm/eagle-public/values-dev.yaml \
  --set image.tag=v1.1.0 \
  --wait

  2. Fix the issue and redeploy - Push fix to develop, which triggers new dev deploy

Preventive Measures

To reduce the need for rollbacks:

  1. Thorough testing in test environment before prod promotion
  2. Review Trivy scan results - Security issues caught early
  3. Use feature flags - Disable problematic features without rollback
  4. Staged rollouts - Consider canary deployments for major changes
  5. Monitoring - Set up alerts to detect issues quickly

Related Documentation