Reverting a release - bcgov/SIMS GitHub Wiki

Deploying applications in containerized environments like OpenShift provides flexibility and scalability, allowing developers to streamline their release processes. However, sometimes it becomes necessary to revert a release due to issues or unexpected behavior. Reverting a release involves rolling back both the application code and any associated database migrations.

Prerequisites

  1. Access to the OpenShift cluster.
  2. Familiarity with OpenShift CLI (oc) and database management tools.
  3. Knowledge of the application's release history and associated database migration scripts.

Procedure

  1. Assess the need for reverting:

    • Identify the reason for reverting the release, such as critical bugs, data integrity issues, or performance degradation.
    • Determine the impact of the rollback on users and other dependent systems.
    • Analyze whether a rollback is the best course of action or if there are alternative solutions.

  2. Review the release history:

    • Identify the version or release point that needs to be reverted.
    • Gather information about the application's code changes, database migrations, and associated release artifacts.

  3. Prepare a rollback plan:

    • Communicate with stakeholders, including developers, operations teams, and any affected users.
    • Define a rollback strategy, considering the impact and urgency of the issue.
    • Ensure you have backups of any critical data or configuration files that might be affected during the rollback.

  4. Stop all the application in openshift:

    • Using the OpenShift CLI (oc), identify the deployment associated with the release to be reverted.
    • Login to Openshift
    • Select the appropriate project to revert
    • Stop all the pods below, by clicking and bringing down to zero
      • prod-api-sims
      • prod-web-sims
      • prod-queue-consumers
      • prod-workers

  5. Revert the database migrations:

    • Identify the database migration scripts associated with the release that needs to be reverted, particularly the number of scripts that ran for the release.
    • Utilize the github action revert-db-migraion to do the migration revert (TODO, need to be created) to apply the appropriate down migration scripts or revert changes made by the release.
    • Ensure the database schema is consistent with the previous version before proceeding.

  6. Revert the release:

    • Use the github action - (Deploy to OpenShift/Camunda) to deploy all the application in the openshift environment.
    • Click on Run workflow and select the values before the workflow run
      • Branch - Release
      • Git Ref (e.g. v1.0.0 or v1.0.0-prerelease.1) - The release to be reverted
      • Environment - Prod / appropriate environment
      • Deploy Camunda BPMN/DMN from main - Checked
      • Deploy Form.io forms from main - Checked
  • Click the button "Run workflow"
  • Login to Openshift go to the appropriate project and check if the pods are deploying with the required number of pods.

  1. Validate the rollback:

    • Thoroughly test the application to ensure the rollback has resolved the issue or restored the expected behavior.
    • Validate data integrity and consistency in the database after applying the down migrations.
    • Involve QA and user acceptance testing (UAT) teams to verify the application's functionality.

  2. Post-rollback tasks:

    • Communicate the successful rollback to relevant stakeholders.
    • Document the reasons for the rollback, steps taken, and lessons learned for future reference.
    • Update the release plan and consider any necessary changes to prevent similar issues in future deployments.

Conclusion

Reverting a release in an OpenShift environment, which includes both application code and database migrations, requires a careful and systematic approach. By following the steps outlined in this write-up, you can ensure a smooth rollback process and minimize disruption to your application and its associated data. Regular backups, effective communication, and thorough testing are essential elements to successfully revert a release.