Emerald Cluster - bcgov/nr-compliance-enforcement-cm GitHub Wiki

Case Management Repo

The case management repo (this repo [https://github.com/bcgov/nr-compliance-enforcement-cm/]) contains the code for the case management application. This repo also contains the following github actions:

  • code quality checks
  • build and push images to GHCR (which results in images being pushed to artifactory)
  • Push of the helm values.yaml file to the gitops repo (to trigger deployments to OpenShift)

Artifactory:

Artifactory is where are application images are stored. We need to store the images here because Argo can't access GHCR.

        docker pull --platform linux/amd64 artifacts.developer.gov.bc.ca/github-docker-remote/bcgov/nr-compliance-enforcement-cm/frontend:latest
        docker pull --platform linux/amd64 [artifacts.developer.gov.bc.ca/github-docker-remote/bcgov/nr-compliance-enforcement-cm/backend:latest](http://artifacts.developer.gov.bc.ca/github-docker-remote/bcgov/nr-compliance-enforcement-cm/frontend:latest)
        docker pull --platform linux/amd64 [artifacts.developer.gov.bc.ca/github-docker-remote/bcgov/nr-compliance-enforcement-cm/database:latest](http://artifacts.developer.gov.bc.ca/github-docker-remote/bcgov/nr-compliance-enforcement-cm/frontend:latest)
        docker pull --platform linux/amd64 [artifacts.developer.gov.bc.ca/github-docker-remote/bcgov/nr-compliance-enforcement-cm/migrations:latest](http://artifacts.developer.gov.bc.ca/github-docker-remote/bcgov/nr-compliance-enforcement-cm/frontend:latest)

Gitops Repo

The gitops repo contains the helm charts to deploy the application to OpenShift. This is required because Argo (used for our deployment pipeline) only has access to this repo (not the case management repo). A github action has been setup in the case management repo to update the values file, which will trigger a deployment of the case management application to Emerald.

URL: https://github.com/[bcgov-c/tenant-gitops-dc0a4a](https://github.com/bcgov-c/tenant-gitops-dc0a4a

Argo

Argo is the gitops component of our pipeline. Once a pull request is approved in the gitops repo, Argo will see the change and deploy the new images to Emerald.

I had to login using my github account to setup argo. All that is required on this end is to create an application. Here's my yaml for that:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: dc0a4a-compenf-dev
spec:
  destination:
    name: ''
    namespace: dc0a4a-dev
    server: 'https://kubernetes.default.svc'
  source:
    path: charts/nr-compliance-enforcement-cm
    repoURL: '[email protected]:bcgov-c/tenant-gitops-dc0a4a'
    targetRevision: update-helm-chart-18
  sources: []
  project: dc0a4a

Adding New Argo Users

BCGov's ArgoCD UI references a CRD for its permission set, in order to add new users the CRD must be editted to contain their github / gov email.

Example for viewing the resource contents: oc get gitopsteam/gitopsteam-compliance-enforcement-cm -n dc0a4a-tools -oyaml

You can use oc edit to directly modify the CRD and save it. We currently do not version control this CRD but plan to in the future.

Deploying New Infrastructure

This assumes you've already added the prerequisite GitHub action job changes to build and push tagged images to GHCR in your app repo

Currently we do not have a guardrails-protected way to stage infrastructure changes, although we will add this in the future

No guardrails implications:

  • If faulty templates are pushed to the tenant repo, ALL environments immediately detect them, and will enter a degraded status and stop updating until the templates are fixed
  • New infrastructure will immediately try to roll to ALL environments unless proper templating control flows are used. See https://helm.sh/docs/chart_template_guide/control_structures/
  • Argo reads from main as its ref, so all infra changes need to merge to main to have any effect, including PR-based environments

Until we have guardrails, here's the proposed flow:

newService:
  enabled: false
  deploymentStrategy: Recreate
  autoscaling:
    enabled: false
    minReplicas: 1
    maxReplicas: 1
    targetCPUUtilizationPercentage: 80
  #-- the service for the component. for inter namespace communication, use the service name as the hostname.
  service:
    type: ClusterIP

It is critical you update the first line of each template yaml file to refer to your new service's enabled flag, otherwise your workload will rollout across all environments, including prod

For example: change {{- if and .Values.global.enabled .Values.backend.enabled }} to {{- if and .Values.global.enabled .Values.newService.enabled }}

Ensure that these are updated FOR EACH template file in your new template folder

  • Before you commit your changes you should verify the templates are valid when populated by a values file, eg:
~/projects/tenant-gitops-dc0a4a/charts/apps$ helm template . --values values/.outputs/values-static.yaml
  • Helm will produce errors you need to fix, otherwise the resulting yaml will be produced to your terminal indicating your templates are valid

Note that helm template contains many flags you can use to experiment with the output of your templates, like the release name

Experimenting with deploying the new service can be done in 2 ways:

GitOps Tenant Repo Method

Argo Override Method

Or force your new service's enabled flag to true using Argo's value override feature. image

  • Click the left-most item, this represents the overall Argo Application configuration managing the dev environment
  • Click the Manifest tab in the drawer that opens
  • Under parameters add the following stanza:
      - name: newService.enabled
        value: true

Argo should now see the templates for your new service and roll it out.

Adding New Secrets

Currently without mounted vault secrets, our flow for secrets is obtuse. We will migrate to vault-managed secrets soon

New secret changes effect all environments at once, including prod UNLESS you add conditional if's around the new changes. Refer to the example in the adding infra section

Modify https://github.com/bcgov-c/tenant-gitops-dc0a4a/blob/main/charts/apps/templates/secret.yaml with your new secret value. If the value is supposed to be something like a unique password credential, set a placeholder string. This will be overridden in Argo.

Don't forget to verify the secret changes with helm template !

Before merging in your secret changes to main, prepare the override by opening the argo application, eg: https://gitops-shared.apps.emerald.devops.gov.bc.ca/applications/openshift-bcgov-gitops-shared/dc0a4a-compenf-dev-0?resource=, navigate to the Manifest tab in the drawer. You will see existing overrides for database passwords. Add your's in a similar fashion.

Currently, every environment needs to manually have its manifest editted to contain the new value overrides

Finally, merge in your secret changes. If everything goes smoothly you can gradually roll out the secret by adjusting the if conditional guards

OpenShift

URLS:

Setup notes

Steps Taken:

Create Artifactory Repo (refer to documentation above in Artifactory section)
Create ArgoCD to create github-c (gitops) repo for manifests (refer to documentation above in Argo section)
Update pipeline to push helm-charts to gitops repo (tenant-gitops-dc0a4a)
Create application in ArgoCD
Create network policy to allow traffic from all pods in TOOLS to bcgov.github.io