DevOps - bcgov/PIMS GitHub Wiki

DevOps Continuous Integration / Continuous Deployment

First it was Waterfall, next it was Agile, and now it's DevOps. This is how modern developers approach building great products. With the rise of DevOps has come the new methods of Continuous Integration, Continuous Delivery, (CI/CD) and Continuous Deployment. Conventional software development and delivery methods are rapidly becoming obsolete. Historically, in the agile age, most companies would deploy/ship software in monthly, quarterly, bi-annual, or even annual releases. Now we build and deploy multiple times a day.

Continuous integration focuses on blending the work products of individual developers together into a repository. Often, this is done several times each day, and the primary purpose is to enable early detection of integration bugs, which should eventually result in tighter cohesion and more development collaboration. The aim of continuous delivery is to minimize the friction points that are inherent in the deployment or release processes. Typically, the implementation involves automating each of the steps for build deployments such that a safe code release can be done—ideally—at any moment in time. Continuous deployment is a higher degree of automation, in which a build/deployment occurs automatically whenever a major change is made to the code.

PIMS CI/CD

The PIMS project currently uses GitHub Actions and OpenShift to support CI/CD.

The general high-level workflow is as follows;

  1. Fork Repo
  2. Clone Repo
  3. Submit PR to main branch
  4. GitHub Action - Run Unit Tests
  5. GitHub Action - Run Code Coverage (CodeCov)
  6. GitHub Action - Build and Push Image to Imagesteam(Pull Request Number used as the Tag for the Image Build)
  7. Merge PR into main branch
  8. GitHub Action - Run Deployement to DEV environment (deployes the image tag (PR number))
    • GitHub Action - Run OpenShift - Deployment Configuration ( API or APP based on the changes )
    • GitHub Action - Run OpenShift - Deployment Configuration DB Migration (if needed)
    • OpenShift - Orchestrate Pods
    • OpenShift - Orchestrate Containers
    • OpenShift - Orchestrate Storage
    • OpenShift - Orchestrate Routes

Environments

There are four projects within OpenShift that are named, TOOLS, DEV, TEST, PROD. Three of which represent the environments where PIMS instances will reside. The fourth is TOOLS which hosts tooling such as Jenkins, SonarQube and others.

Environments

Name URL Description
DEV pims-dev.apps.silver.devops.gov.bc.ca The development environment for Developers
TEST pims-test.apps.silver.devops.gov.bc.ca The testing environment for QA and UAT
PROD pims.gov.bc.ca The production environment

Branching

Repository branches are used to manage development. There is only one branch main.

Branch Description
main Default branch for all environments
{pims-####} These branches are normally hosted in forked repositories and link to Jira Stories.

It's essential to note that deployments to DEV, TEST, and PROD environments are not dictated by separate branches. Instead, configurations rely on image tags for control and versioning.

DEV Environment

Deploying changes to the DEV environment involves a straightforward automated process. The workflows are named as follows:

  • API Image Build: DEV API: Image Build on PR
  • APP Image Build: DEV APP: Image Build on PR
  • API Deployment: DEV API: Image Deploy
  • APP Deployment: DEV APP: Image Deploy

Developers initiate the deployment by creating a pull request (PR) to merge their changes into the main branch. This action then triggers the GitHub workflow, specifically either the DEV API-:Image Build on PR or DEV APP-:Image Build on PR GitHub Action workflow, or both, depending on the nature of the file changes.

Once the PR is approved and ready to merge, the developer will merge the PR. The merging process automatically triggers the GitHub Action, further promoting the image tag in the DEV environment. This streamlined process ensures that the changes are promptly deployed to the DEV environment for initial quality assurance (QA) testing.

Note: To ensure that the build image tag consistently reflects the latest code from the main branch along with any new work, it is mandatory to perform an "Update branch" before merging. The "Update branch" process involves merging the main branch into the temporary branch intended for merging (dev), ensuring that images are updated with the most recent code changes. This practice guarantees that the build image accurately represents the current state of the codebase before any merging occurs.

TEST Environment Deployment

For changes to be promoted to the TEST environment, a manual trigger of the GitHub Action is essential. The workflows are named as follows:

  • API Deployment: TEST API: Image Deploy
  • APP Deployment: TEST APP: Image Deploy

API Deployment Workflow

The GitHub Action workflow for API deployment requires two inputs:

  1. API Image Tag: This tag specifies the version of the API image to be deployed.
  2. Migration Image Tag: Optional input; only required if there is a need to promote a migration image.

By providing these inputs, the GitHub Action orchestrates the deployment of the specified image tags to the TEST environment. The inclusion of a migration image tag is necessary only when migration image promotion is required.

APP Deployment Workflow

For APP changes, the GitHub Action workflow requires the following input:

  • APP Image Tag: This tag is mandatory to initiate the workflow.

By specifying the APP image tag, the GitHub Action facilitates the deployment of the corresponding image to the TEST environment. This manual trigger and input-based approach ensure controlled and deliberate promotion of changes in the testing environment.

PROD Environment Deployment

Similar to TEST, for PROD as well, a manual trigger of the GitHub Action is essential. The workflows are named as follows:

  • API Deployment: PROD API: Image Deploy
  • APP Deployment: PROD APP: Image Deploy

The GitHub Action workflow for API deployment in the PROD environment follows a similar pattern. It entails providing two inputs:

  1. API Image Tag: This tag designates the version of the API image intended for deployment.
  2. Migration Image Tag: An optional input, only necessary if there is a requirement to promote a migration image.

Similarly, the workflow for APP changes in the PROD environment adheres to a straightforward input-based process:

  • APP Image Tag: This mandatory tag initiates the workflow, guiding the GitHub Action to facilitate the deployment of the corresponding image to the PROD environment.

CI/CD Pipelines

In the CI/CD Pipelines Overview table below, it present a comprehensive summary of the key pipelines involved in our development lifecycle. These pipelines, designed for the DEV, TEST, and PROD environments, encompass image builds and deployments, both automated and manual.

Name Destination Trigger Description
DEV API: Image Build on PR Openshift Imagestream Auto Builds the API Image upon PR creation, incorporating the PR number as the tag.
DEV APP: Image Build on PR Openshift Imagestream Auto Constructs the APP Image during PR creation, utilizing the PR number as the tag.
DEV API-Image Deploy on PR merge DEV Auto/Manual Automatically deploys the API Image upon PR merge, utilizing the PR number as the tag. Additionally, can be manually triggered for rollback purposes.
DEV APP-Image Deploy on PR merge DEV Auto/Manual Automatically deploys the APP Image upon PR merge, using the PR number as the tag. Also, can be manually triggered for rollback purposes.
TEST API-Image Deploy TEST Manual Manually initiates the pipeline to deploy the API Image, providing the API image tag and Migration Image tag (if needed).
TEST APP-Image Deploy TEST Manual Manually initiates the pipeline to deploy the APP Image, specifying the API image tag.
PROD API-Image Deploy PROD Manual Manually initiates the pipeline to deploy the API Image, providing the API image tag and Migration Image tag (if needed).
PROD APP-Image Deploy PROD Manual Manually initiates the pipeline to deploy the APP Image, specifying the API image tag.

Image Tagging Strategy

In our CI/CD pipeline, image tagging is a crucial aspect that facilitates version tracking, feature identification, and deployment to specific environments within OpenShift. Our tagging strategy revolves around associating images with their corresponding PR numbers, creating a clear linkage between images and the development lifecycle.

  1. Tagging Trigger: Images are built and tagged automatically when a new PR is created, triggered by changes in files. The tagging process differentiates between API and App changes, ensuring precision in versioning.

  2. Database Migration: The tagging strategy includes tying the Database migration image to API changes. This image is built and tagged only if there is a modification in the Migration folder, streamlining the tagging process for database-related alterations.

Benefits:

  • Traceability: Each image is distinctly tied to a PR number, providing a traceable and identifiable link to the corresponding development work.

  • Prevention of Broken Code: Image builds occur exclusively during PR creation, ensuring that only validated code is merged into the Main branch. This safeguards against the introduction of broken code.

  • Rollback Capability: The tagging strategy enables a straightforward rollback to a known and stable image tag. In case of issues or errors, reverting to a previous image tag is easily achievable, minimizing downtime and facilitating quick issue resolution.

  • Enhanced Readability: PR-based tagging enhances the readability of the image tags. Each tag reflects the specific PR number, making it intuitive and straightforward to understand the context and purpose of each image.

Important Consideration:

  • Non-Sequential Tags: It's essential to note that PR tags are not sequential. For instance, Tag 1921 is not a successor of 1920. If 1920 is merged after 1921, Tag 1920 will incorporate the changes for 1921. This emphasizes the mandatory "Update branch" step, ensuring the branch is updated before any merging occurs.

Development Releases

Development Workflow:

  1. Developers initiate the deployment process by creating a pull request (PR) to merge their changes into the main branch.

  2. The CI/CD pipeline triggers specific workflows, such as DEV API: Image Build on PR and DEV APP: Image Build on PR, automatically building and tagging API and APP images using the PR number.

Database Migration Integration:

  1. The migration image is built and tagged only if there is a modification in the Migration folder during the PR.

Automated Deployment on PR Merge:

  1. Upon approval and readiness, developers merge the PR into the main branch.

  2. The GitHub Action workflow (DEV API-Image Deploy on PR merge and DEV APP-Image Deploy on PR merge) is automatically triggered, deploying the API and APP images to the DEV environment using the PR number as the tag.

Below is attached a diagram explaining how API changes would deploy using PR as tags.

image

Production Releases

Every Sprint, a new releasable increment is developed, adhering to our versioning strategy (1.0.0 = [major.minor.patch]).

Once a releasable increment is deemed ready, the changes can be promoted to the TEST environment for thorough testing and quality assurance. To initiate this promotion to TEST, developers can trigger the GitHub Action pipeline, specifying the desired version tag they want to promote. During the pipeline execution, a prompt will request a tag name. For instance, if developers wish to proceed with the changes deployed in the DEV environment, they can use the tags from the Image Tags Wiki page table to guide the promotion further. Below is an example of the input UI from the GitHub Action that represents the input field for the API test deployment workflow:

image

image

Similarly, for the production release, the same steps would be followed by providing the tags currently working in the TEST environment.

Upon successful QA and UAT approval in the TEST environment, a manually triggered production pipeline can be initiated. This production pipeline will prompt for the appropriate tag to release. Developers should input the tag that corresponds to the version that has been thoroughly tested and approved in the TEST environment. This ensures a controlled and deliberate promotion of changes from the TEST environment to the production environment.

Rollback

In the event of a failure during the deployment phase, it becomes necessary to perform a rollback of the release images that were deployed and restore the database backup. Developers have the flexibility to rollback to a previous version at any given point by utilizing tags in each environment.

This comprehensive rollback strategy ensures that developers can quickly and manually initiate a rollback to a known and stable version in case of any deployment issues.