OpenShift - bcgov/PIMS GitHub Wiki

Pipelines

• DEV Pipeline
• TEST Pipeline
• PROD Pipeline

Builds

• BuildConfigs
• ImageStreams - Including tagging images

Renewing an SSL Cert

Testing the Production Build Locally

Database Backups

• Backups
• Restore
• Changing Configuration Settings

"Last Editied By": "Brady Mitchell"
"Date": "June 2nd, 2023"

pims-pipeline

• Builds and deploys to DEV.
• Automatically triggered when merging to dev branch.

1. A pull request is merged to the dev branch.
2. Currently, there is two webhooks in the GitHub repo that trigger when a push or pull request are made. These are http://pims-tekton.apps.silver.devops.gov.bc.ca/hooks/api and http://pims-tekton.apps.silver.devops.gov.bc.ca/hooks/app.
3. In OpenShift, the webhooks can be found under Networking > Routes as git-webhook-pim-app and git-webhook-pim-api.
4. There, routes are connected with services el-git-pims-app and el-git-pims-api which trigger the pipeline event listener.
5. The pipeline event listeners can be found in Pipelines > Triggers > Event Listeners as git-pims-app and git-pims-api. These triggers will run the pipeline trigger templates git-pims-app and git-pims-api if changes are made to the dev branch in either frontend or backend.

Tasks/Steps:

app-build - Builds from the pims-app-base.dev BuildConfig and then pims-app.dev BuildConfig.

"Last Editied By": "Brady Mitchell"
"Date": "May 10th, 2023"

master-pipeline

• Builds and deploys to TEST.
• Manually triggered by selecting the pipeline, clicking Actions > Start, and then specifying an image tag such as v02.03.12.
• To know what image tag to use, follow these steps:
  • Go to Builds > Image Streams, find and click on pims-app or pims-api.
  • Scroll to the bottom of the Tags section.
  • Under Name column (1st column on left), you should see the final entry looks something like pims-app:v02.03.12.
  • The image tag to use when running the pipeline should be above this version number such as v02.03.13.
  • Use semantic versioning:
    • Major overhall to the application: v03.00.00
    • Minor but significant changes to the application: v02.04.00
    • Patch or minor changes to the application: v02.03.13

"Last Editied By": "Brady Mitchell"
"Date": "May 10th, 2023"

deploy

• Deploys to PROD.
• Manually triggered by selecting the pipeline, clicking Actions > Start, and then specifying an image tag such as v02.03.12.
• To know what image tag to use, follow these steps:
  • Go to Builds > Image Streams, find and click on pims-app or pims-api.
  • Scroll to the bottom of the Tags section.
  • Under Name column (1st column on left), you should see the final entry looks something like pims-app:v02.03.12.
  • Use this image tag, example: v02.03.12.

Return To Top

"Last Editied By": "Brady Mitchell"
"Date": "June 2nd, 2023"

pims-app-base.dev :

• References frontend/Dockerfile.ocp which uses vite to build the production code.
• This Dockerfile references a nodejs image that can be found in ImageStreams.
• Outputs to pims-app-base:latest image.
• This output image will then be used by pims-app.dev during the app-build task/step of the pims-pipeline.
• To test production build locally, reference Testing the Production Build Locally

pims-app.dev :

• Builds nginx runtime based on pims-app-base:latest image.
• Outputs to pims-app:latest image.

pims-app-base.master :

• Specifies the source image for building the TEST app.
• Outputs to pims-app-base:latest image.

pims-app.master :

• Builds nginx runtime based on pims-app-base:latest image.
• Outputs to pims-app:latest image.

pims-api.dev :

• Specifies Dockerfile with image for building DEV api. This image in the Dockerfile.ocp is referencing an image in OpenShift tools namespace.

pims-api.master :

• Specifies Dockerfile with image for building TEST api. This image in the Dockerfile.ocp is referencing an image in OpenShift tools namespace.

pims-database.dev :

• Specifies Dockerfile with image for building DEV database.

pims-database.master :

• Specifies Dockerfile with image for building TEST database.

backup-mssql :

• Specifies the Docker image to use for the mssql database backups. In the yaml file of this build config, the "dockerStrategy" section contains the url to the image that will be used during the build:

    `kind: DockerImage`
  
    `name: 'mcr.microsoft.com/mssql/rhel/server:2022-latest'`
  

• This "name" value can be updated whenever the backup-mssql image needs to be updated. After saving changes to the yaml file, all you need to do is start the "build" by clicking the actions tab in the top right of the backup-mssql buildconfig and then select "Start build". Once this completes the 'backup-mssql' image latest tag should be updated with the mssql image that you used in the yaml file.

"Last Editied By": "Sharala Perumal"
"Date": "Sept 15th, 2023"

Tagging Images:

• In OpenShift, copy login command from the account dropdown in the top right corner of the dashboard.
• In a terminal with oc CLI tool installed, paste and run the login command.
• Make sure you are in the tools namespace by executing oc project 354028-tools.
• Tag an image using oc tag <image> <imagestream>:<tag>
  • Where <image> is the image and tag you want to use, such as a Docker image,
    and <imagestream>:<tag> corresponds with an image in the Builds > ImageStreams section of OpenShift.
  • Example: oc tag docker.io/node:18.17.1-bullseye-slim nodejs:18.17.1-bullseye-slim.
  • Note: For docker images you may need to first docker pull <image> and then oc tag docker.io/<image> <imagestream>:<tag>.

nodejs :

• NodeJS image for building the frontend app.
• Has tag 18.17.1-bullseye-slim used by frontend/Dockerfile.ocp

dotnet-aspnet-runtime :

• DotNet Runtime image from Microsoft on Docker Hub for running backend api.
• Latest used tag is 7.0.

dotnet-sdk :

• DotNet SDK image from Microsoft on Docker Hub for building and compiling backend api.
• Latest used tag is 7.0.

mssql-rhel-server :

• Microsoft SQL Server from Microsoft on Docker Hub for running the database.
• Latest used tag is 2022.

backup-mssql :

• MSSQL backups using plugin developed by https://github.com/BCDevOps/backup-container
• Latest used tag is latest.
• To update the latest image, there is a "buildconfig" called backup-mssql in the tools namespace in which you need to edit the yaml section for git ref which refers to which version of the backup container to use. (see build config section for backup-mssql above)
• After updating the latest tag, to update the remaining dev, test, and prod tags for this image, just run the following commands: oc tag -n 354028-tools backup-mssql:latest backup-mssql:dev, oc tag -n 354028-tools backup-mssql:latest backup-mssql:test, and oc tag -n 354028-tools backup-mssql:latest backup-mssql:prod

Return To Top

"Last Editied By": "Lawrence Lau"
"Date": "April, 2023"

• Before the SSL certificate for the production site expires (usually a month in advance), we should get contacted by someone at IMB operations who will send us three files for renewing the certificate: pims.gov.bc.ca.txt, pims.gov.bc.ca.key, and L1KChain.txt.
• Basically there are 2 routes in the production namespace: pims-api & pims-app that we will need to "modify" in order to update the 3 fields corresponding to the Certificate, CA Certificate, and Key fields that are contained within the yaml of the routes.
• What we will be doing in the next steps will be to download the existing routes for the app and the api to our local machines so that we can update those 3 fields mentioned above with the new files that contain the new public and private keys. The final steps will be to "apply" the new route which will update the 2 routes in our production environment.

• Login to your Openshift project using the oc cli, and download a copy of the "pims-app" and the "pims-api" routes yaml files from the prod namespace as a backup.
• Then use the command below to create a copy of the route on your local machine which will be used when applying the new cert. The command below will create 2 new files called pims-api-2022.yaml and pims-app-2022.yaml, based on the existing pims-api and pims-app routes, in your current directory which in our case was the root directory, the -o yaml outputs the yaml of the route into a new file which you can name whatever you want, in this case it's named: pims-app-2022.yaml. Once this file is created, we will need to "clean up" and get rid of some unnecessary fields.
  • oc get route pims-api -o yaml > pims-api-2022.yaml
  • oc get route pims-app -o yaml > pims-app-2022.yaml
• Next open the newly created file/s in the previous step and delete the unnecessary fields and sections in the yaml files: uid, resourceVersion, creationTimestamp, annotations, managedFields section (this one is long), status section (at the bottom). These fields will be automatically generated when the route gets “re-applied”.
• When replacing the new caCertificate, certificate, and key fields you can use this syntax shown below and then copy and paste the entire certificate/s into each section. The pipe operator with the dash |- is yaml syntax which recognizes a multi-line string, so you don’t need to convert the certificate into one long string.

   caCertificate: |-
      -----BEGIN CERTIFICATE-----
      MIIFDjCCA/agAwIBAgIMDulMwwAAAABR03eFMA0GCSqGSIb3DQEBCwUAMIG+MQsw
      CQYDVQQGEwJVUzEWMBQGA1UEChMNRW50cnVzdCwgSW5jLjEoMCYGA1UECxMfU2Vl
      IHd3dy5lbnRydXN0Lm5ldC9sZWdhbC10ZXJtczE5MDcGA1UECxMwKGMpIDIwMDkg
      ...

• Once the new certs have been copied and pasted into the yaml file, save it, then do a dry run using the following command:

  • oc apply -f pims-api-2022.yaml --dry-run=client

• If the dry run was successfully, you should see the following with no errors:

  • route.route.openshift.io/pims-api configured (dry run)

• As long as there are no errors, you should be able to “apply” the new yaml which will update the route in openshift:

  • oc apply -f pims-api-2022.yaml

• If the apply was successful you will see an output similar to below:

  • route.route.openshift.io/pims-api configured

• You can then follow the same steps to apply the updated yaml to the pims-app

  • oc apply -f pims-app-2022.yaml --dry-run=client

• If the dry run was successfully, you should see the following with no errors:

  • route.route.openshift.io/pims-app configured (dry run)

• As long as there are no errors, you should be able to “apply” the new yaml which will update the route in openshift:

  • oc apply -f pims-app-2022.yaml

• If the apply was successful you will see an output similar to below:

  • route.route.openshift.io/pims-app configured

Return To Top

"Last Editied By": "Brady Mitchell"
"Date": "June 15th, 2023"

To test the production build (as it would be in OpenShift) on your local machine, change target: dev to target: prod under the frontend service in docker-compose.yml. Run make npm-refresh or make rebuild n=frontend to rebuild the frontend.

Return To Top

"Last Editied By": "Lawrence Lau"
"Date": "June, 2023"

Database backups are created every morning at 1am in Dev, Test, and in Prod. There are daily, weekly, and monthly backups, and only the most recent 3 daily backups, 1 weekly, and 1 monthly backup are retained. In each of the Dev, Test, and Prod environments there is a "pims-backup" pod that you can access to either view the backups or to manually restore a specific backup for the PIMS database.

To view existing backups, click on the Terminal tab of the pims-backup pod, and then in the terminal type the following command: ./backup.sh -l

You can view the configuration settings for the backup by clicking on the "Logs" tab:

A weekly backup should usually be created every Sunday at 1am, and the monthly backups should be created on the last day of the month. All older backups will be automatically pruned.

To restore a specific backup, open the terminal for in the environment which you want to restore, and run the following command:

./backup.sh -r mssql=pims-database:1433/pims -f /backups/monthly/2023-05-31/pims-database-pims_2023-05-31_13-55-43

You can also run ./backup.sh -h to get a list of all the commands that you can run. Also there is more documentation on the backup container itself: https://github.com/BCDevOps/backup-container as well as some additional documentation here: https://github.com/bcgov/PIMS/blob/dev/maintenance/RESTORE_BACKUPS.md and here: https://github.com/bcgov/PIMS/blob/dev/maintenance/BACKUP.md

You can change the configuration of the pims-backup by modifying the pims-backup DeploymnentConfig "Environment" variables in Openshift. For example, you can set the number of daily (DAILY_BACKUPS), weekly (WEEKLY_BACKUPS), and monthly (MONTHLY_BACKUPS) backups in the deployment config as well as the total number of backups to keep (NUM_BACKUPS is optional).

Once you save the deployment config, it will terminate the existing pims-backup pod and create a new one with the new configuration changes.

To set the "create a Rocketchat integration you can set the WEBHOOK_URL variable, but first you will need to create a Key/Value secret to store the rocketchat url in Openshift:

and then in the "pims-backup" deployment config you would choose that secret with the corresponding "key" to use:

After you save the "pims-backup" deployment config, it will create a new "pims-backup" pod, which you can then use the terminal tab in the pod to check the settings by viewing the logs tab:

Note that any change to the "pims-backup" deployment config will remove the "pims-"backup" pod and will create a new pod.

Return To Top