Home - michaelsauter/ods-pipeline GitHub Wiki

ODS provides CI/CD pipeline support based on OpenShift Pipelines. This introduction will walk you through the essentials, and guide you all the way to more advanced topics. Basic knowledge of Kubernetes concepts and OpenShift is assumed. Estimated reading time is about 15 minutes.

OpenShift Pipelines is a Kubernetes-style CI/CD solution based on Tekton. It builds on the Tekton building blocks and offers tight integration with OpenShift. The main addition over plain Tekton is a UI in the OpenShift console.

Tekton provides a framework to create cloud-native CI/CD pipelines. The two main components of Tekton are:

• Tekton Pipelines - this is the foundation of Tekton. It defines a set of Kubernetes Custom Resources that act as building blocks from which CI/CD pipelines can be assembled.

• Tekton Triggers - this allows to instantiate pipelines based on events. For example, one can trigger the execution of a pipeline every time a PR is merged against a GitHub repository.

We'll look at Tekton Pipelines in detail now, and will look at Tekton Triggers later on in this introduction.

A Tekton pipeline (a Kubernetes resource named Pipeline) references a series of tasks (a Kubernetes resource named Task). When the pipeline runs, Kubernetes will schedule one pod per task. Each task is made up of a series of steps. Each step corresponds to one container in the task pod. At a minimum, a step defines the container image to use, and which command / script to run. Therefore, a step can achieve a huge variety of things such as building artefacts, deploying, etc. Tekton distinguishes between the definition (Pipeline and Task resources) and the actual execution (also modelled as resources, namely PipelineRun and TaskRun). The PipelineRun provides a workspace to the pipeline, which is a Kubernetes volume mounted in the task pods. If the volume is a PVC, it can be shared between tasks, allowing the tasks to work e.g. on the same repository checkout. The following illustrates the basic Tekton architecture:

At this stage you know just enough about Tekton to continue with this introduction, but if you want to know more about it, you can read the Tekton docs and/or follow the OpenShift Pipelines tutorial.

In regards to CI/CD, ODS provides two things:

• a few Tekton Tasks
• a webhook interceptor

We'll look at the Tekton tasks now and come back to the webhook interceptor later as it is connected to Tekton Triggers, the second main component of Tekton.

The ODS tasks can be used in a pipeline to build, deploy and test your application. Note that ODS does not implement it's own CI/CD system: The tasks provided by ODS are regular Tekton tasks and in fact you can use any Tekton task in a pipeline in addition to or instead of the tasks provided by ODS.

The tasks are so easy to exchange and compose as Tekton tasks have clearly defined inputs (the parameters), clearly defined outputs (the results) and work on a generic workspace, for which an actual volume is provided to them by the pipeline.

An ODS installation provides you with the following tasks, which are implemented as ClusterTask resources, allowing you to use them in any namespace:

• ods-start: Checkout repository and set Bitbucket build status
• ods-build-go: Build a Go application (includes Sonar scan)
• ods-build-java: Build a Java application (includes Sonar scan)
• ods-build-python: Build a Python application (includes Sonar scan)
• ods-build-nodejs: Build a TypeScript/JavaScript application (includes Sonar scan)
• ods-build-image: Package application into container image (includes Aqua scan)
• ods-deploy-helm: Deploy Helm chart
• ods-test-geb: Run Geb/Spock tests
• ods-test-cypress: Run Cypress tests
• ods-finish: Set Bitbucket build status

Let's look at the ods-build-* tasks in more detail to understand what such tasks provide. The ods-build-go tasks consist of the following steps:

• Build Go binary (through running go build, go test, golangci-lint run etc.)
• Scan binary with Sonar

The other "language build tasks" like ods-build-java have the same steps, except that they make use of e.g. gradle build to build a JAR instead of a Go binary.

The ods-build-image tasks consist of the following steps:

• Build container image with Buildah
• Scan image with Trivy / Aqua
• Push container image to image stream

The produced images are tagged with the Git commit SHA being built. If the task detects this tag to be already present in the image stream, all steps are skipped.

The behaviour of each task can be customized by setting parameters. For example, the ods-build-image tasks assumes the Dockerfile to be located in the docker directory by default. You can instruct the task to use a different Docker context by providing the context-dir parameter to the task.

As you have learned earlier, tasks are referenced by pipelines. Therefore, all you would need to do to use the ODS tasks is to create a pipeline in OpenShift, and reference the tasks you want to execute. Then you'd need to start the pipeline (which creates a PipelineRun).

While using this approach is possible, it has a few drawbacks:

• You would need to create a pipeline for each repository (if they use different tasks or parameters)
• You would need to manage the pipelines in the UI
• You would need to start the pipeline manually after each commit

To solve these problems (and a few more that will become apparent later), ODS ships with another component alluded to earlier, the ODS webhook interceptor. Together with the functionality that Tekton Triggers provides out of the box, this interceptor allows to automate the creation, modification and execution of pipelines based on task definitions stored in the Git repository to which the pipeline corresponds.

To understand how this works, it is best to trace the flow starting from the repository. Assume you have a repository containing a Go application, and you want to run a pipeline building a container image for it every time you push to Bitbucket. To achieve this in a project created by ODS, all you need is to have an ods.yml file in the root of your repository. The ods.yml file defines the tasks you want to run in the pipeline. However, the tasks are not defined as one list, but rather associated with so called phases. ODS views pipelines as a sequence of six phases:

1. Init (e.g. checking out code, notifying systems of pipeline start)
2. Build (e.g. build artefact such as container image)
3. Deploy (e.g. deploy artefact)
4. Test (e.g. run tests against new deployment)
5. Release (e.g. release staged deployment and make it accessible to the public)
6. Finalize (e.g. creating reports, notifying systems of pipeline finish)

Those six phases do not provide logic on their own, instead they just serve as a structure for the pipeline. In fact, they are only a "container" for tasks, and may contain zero or more tasks. We'll come back to the reason for this concept of phases later on. For now, let's look at an example ods.yml file for our Go repository:

phases:

  build:

  - name: backend-build-go
    taskRef:
      kind: ClusterTask
      name: ods-build-go
    workspaces:
    - name: source
      workspace: shared-workspace

  - name: backend-build-image
    taskRef:
      kind: ClusterTask
      name: ods-build-image
    runAfter:
    - backend-build-go
    params:
    - name: image-stream
      value: backend
    workspaces:
    - name: source
      workspace: shared-workspace

  deploy:

  - name: backend-deploy
    taskRef:
      kind: ClusterTask
      name: ods-deploy-helm
    params:
    - name: release-name
      value: backend
    workspaces:
    - name: source
      workspace: shared-workspace

You can see that it defines three tasks: the build phase references the ods-build-go task and the ods-build-image task, and the deploy phase references the ods-deploy-helm task.

In a repository created through ODS quickstarter provisioning, you already have an ods.yml file with task definitions, and when a commit is pushed to Bitbucket, a pipeline reflecting those tasks will start automatically. However, any repository can gain this functionality by adding an ods.yml file and setting a webhook firing on every push in the Bitbucket repository.

The following will describe what happens once this webhook fires. A payload with information about the pushed Git commit is sent to a route connected to an event listener in OpenShift. The event listener is a small service provided by Tekton Triggers, running in your OpenShift namespace. However, before the payload arrives at the event listener, it is sent through interceptors. In the case of an ODS project, two interceptors are configured:

1. A Bitbucket interceptor. This interceptor is provided by Tekton Triggers and checks the authenticity of the request (did the request really originate from a push in the Bitbucket repository?)
2. A custom ODS interceptor.

This custom ODS interceptor is a small service, provided by ODS, and running in your OpenShift namespace. When it receives the request, it retrieves the ods.yml file from the Git repository/ref identified in the payload, and reads the phases configuration. Based on the tasks defined there, it assembles a new Tekton pipeline. The name of this new pipelines is a concatenation of the repository name and the Git ref (e.g. myapp-master). In the next step, the interceptor checks if a pipeline with that name already exists, and either creates a new pipeline or updates an existing pipeline. That way, you get one pipeline per branch which makes it easier to navigate in the OpenShift UI and allows to see pipeline duration trends easily. Finally, the interceptor adds the name of that pipeline to the webhook request payload, and the standard Tekton Triggers flow continues. This means that the amended request is forwarded to the event listener, which handles the event using a trigger binding and a trigger template. The trigger binding extracts values from the request payload to pass as parameters to the pipeline, and the trigger templates instantiates a pipeline run for the pipeline name passed from the ODS interceptor with the parameters provided by the trigger binding.

With the above in place, you do not need to manage pipelines manually. Every repository with an ods.yml file and a webhook configuration automatically manages and triggers pipelines based on the defined tasks.

At this stage you know just enough to get started using and modifying CI/CD pipelines with ODS. Read on for more complex scenarios (e.g. multi-repo pipelines) and some additional considerations.

So far in this introduction, we have dealt with one repository only. However, sometimes an application is made up of several components, each of which is stored in its own repository. In fact, this is the default when you use ODS quickstarter provisioning. While sometimes you want to deploy components individually (pure microservices approach), there are also use cases for which you want to tie all components together in one release. ODS provides support for this as well.

The basic concept is that the ods.yml file can define "child" repositories. The ODS webhook interceptor will then assemble the pipeline based on the tasks defined in each ods.yml file of the referenced child repositories.

As an example, assume you have a TypeScript frontend application next to the Go application example above. Structurally, the TypeScript repository has the same phases/tasks as the Go repository:

phases:

  build:

  - name: frontend-build-nodejs
    taskRef:
      kind: ClusterTask
      name: ods-build-nodejs
    workspaces:
    - name: source
      workspace: shared-workspace

  - name: frontend-build-image
    taskRef:
      kind: ClusterTask
      name: ods-build-image
    runAfter:
    - frontend-build-nodejs
    params:
    - name: image-stream
      value: frontend
    workspaces:
    - name: source
      workspace: shared-workspace

  deploy:

  - name: frontend-deploy
    taskRef:
      kind: ClusterTask
      name: ods-deploy-helm
    params:
    - name: release-name
      value: frontend
    workspaces:
    - name: source
      workspace: shared-workspace

Now you can have a separate "umbrella" repository which combines both components in its ods.yml:

repositories:
- name: backend
- name: frontend

When a pipeline is started for this umbrella repository, it will create a pipeline that has the following task sequence:

1. backend-build-go
2. backend-build-image
3. frontend-build-nodejs
4. frontend-build-image
5. backend-deploy
6. frontend-deploy

The order of the tasks is defined by the order of the phases (init > build > deploy > test > release > finalize), and the tasks in each phase are ordered by the repository order defined in the umbrella repository ods.yml file. It is also possible for the umbrella repository to define its own phases/tasks. They will be added as the last tasks in each phase in the assembled pipeline.

Earlier in this guide, you learned that the ods-build-* tasks skip their steps if the artefact (the container image) which is supposed to be built already exists. Therefore, the multi-repo pipeline will not do any work in tasks one through four. However, the Kubernetes pods are still created, which results in significant overhead. As an optimization, you can use when expressions in the task definition to avoid running tasks. As an example, the frontend-build-image task of the frontend repo would look like this then (see the when at the end):

name: frontend-build-image
taskRef:
  kind: ClusterTask
  name: ods-build-image
params:
- name: image-stream
  value: frontend
workspaces:
- name: source
  workspace: shared-workspace
when:
- input: "$(params.frontend-build-image-identical-taskrun)"
  operator: notin
  values: ["success"]

But what is this <TASKNAME>-identical-taskrun parameter, and how is it set? For this, we need to look at another important building block of ODS pipelines: Nexus, an artefact repository. On project creation, ODS sets up a Nexus repository for each project. All ODS tasks upload any artefacts (such as image SHAs, test reports, etc.) to Nexus at the end of every step. Each artefact is associated with the Git commit SHA. Later tasks (either in the same pipeline or another pipeline) can then discover these artefacts in Nexus, download them, and act based on the provided information. More importantly, at the end of every pipeline (in a Tekton finally task), a "pipeline status" artefact is uploaded to Nexus, recording a status for each task which ran. This artefact is searched for also at the beginning of every pipeline run, and one parameter per task is set to the corresponding value (e.g. <TASKNAME>-identical-taskrun might be success or failure). Therefore, if a pipeline for the same webhook event and Git commit SHA has run already, all tasks with such a when expression are skipped.

If this when guard is added to all build tasks, they will be skipped (no pod will be created) and the run sequence will simply be:

1. backend-deploy
2. frontend-deploy

The deploy tasks still know which images to push into the release namespace because the now-skipped build tasks have uploaded the generated image SHAs in a previous run.

Note that the Nexus repository has a cleanup policy that will delete artefacts if they haven't been downloaded in a certain time period. Otherwise, the Nexus repository would grow forever. If a pipeline for the same commit runs after the cleanup has taken place, the tasks are executed again and the artefacts are being created again.

Further, even though the "pipeline status" artefact (which determines the <TASKNAME>-identical-taskrun parameter) is only created at the end of a pipeline run, there shouldn't occur races between two pipelines (e.g. one for the repo and a multi-repo one) as mounting the same volume ensures the pipelines are being worked on sequentially.

Sometimes you might want to run only parts of the pipeline based on events that occur within the Bitbucket repository. So far we only considered a webhook firing on push (repo:refs_changed), but you can also fire a webhook for other events, such as when a PR is opened (pr:opened). For example, this allows to decorate pull requests with static analysis reports from SonarQube.

You've already came across when expressions in the context of multi-repo pipelines, and the same mechansim is used to run only parts of a pipeline. Each pipeline has a trigger-event parameter that can be used to figure out what action triggered the pipeline. The following phases configuration ensures that the ods-build-go task is executed also for pr:opened events so that pull request decoration can happen, but it does not run the ods-build-image for that event as image building does not need to happen again when a pull request is opened.

phases:

  build:

  - name: backend-build-go
    taskRef:
      kind: ClusterTask
      name: ods-build-go
    workspaces:
    - name: source
      workspace: shared-workspace
    when:
    - input: "$(params.backend-build-go-identical-taskrun)"
      operator: notin
      values: ["success"]
    - input: "$(params.trigger-event)"
      operator: in
      values: ["repo:refs_changed", "pr:opened"]

  - name: backend-build-image
    taskRef:
      kind: ClusterTask
      name: ods-build-image
    runAfter:
    - backend-build-go
    workspaces:
    - name: source
      workspace: shared-workspace
    when:
    - input: "$(params.backend-build-image-identical-taskrun)"
      operator: notin
      values: ["success"]
    - input: "$(params.trigger-event)"
      operator: in
      values: ["repo:refs_changed"]

Without a when expression guarding for certain events, tasks run for all webhook events. Therefore it is recommended to have an explicit when expression for all tasks defining for which events they should run.

This feature is quite powerful: for example you can react to certain PR comments. The text of added PR comments is available to the pipeline in the comment parameter (empty when the event is not for a comment). If you wanted to have a preview deployment happen whenever someone adds a /preview comment, you can configure this:

phases:

  deploy:

  - name: backend-deploy
    taskRef:
      kind: ClusterTask
      name: ods-deploy-helm
    params:
    - name: release-name
      value: backend-preview
    workspaces:
    - name: source
      workspace: shared-workspace
    when:
    - input: "$(params.comment)"
      operator: in
      values: ["/preview"]
    - input: "$(params.trigger-event)"
      operator: in
      values: ["pr:comment:added"]

The ODS pipeline approach was developed with the following use cases in mind. They are extrapolated from existing ODS 3.x quickstarters and gaps identified in ODS 3.x.

• monorepo (one repo defines all components of an app)
• multi-repo (one repo per component, umbrella repo to glue repos together)
• running a sidecar container (e.g. postgres) during a pipeline stage
• one deployment / one image / one container (e.g. Java or Go quickstarter)
• one deployment / multiple images / multi-container (e.g. monorepo quickstarter)
• multiple deployments / multiples images / one container each (e.g. Jupyter quickstarter)
• deploy to OpenShift using Helm
• deploy to AWS or similar
• using solutions like buildpacks vanilla from the internet
• acceptance test repo (e.g. Geb quickstarter)
• partial pipelines (e.g. PR decoration on pr:opened event)

In general, tasks in ods.yml are regular Tekton tasks, and they are added to the assembled pipeline as-is without modification. There are two exceptions to this:

1. runAfter handling. The phases (init, build, etc.) run sequentially and ensure that any runAfter specification does not violate this. If runAfter is not specified in ods.yml, then it is set to the last task of the previous phase. However if it is specified, the ODS webhook interceptor checks that it is referencing a task within the current phase.
2. Workspace handling. Child repositories in multi-repo pipelines will be cloned to <root-repo>/.ods/repos/. Consequently, the workspace configuration of tasks sourced from child repositories will be enhanced with a subPath configuration so that the tasks' working directory points to the root of the child repository. Further, the content umbrella repository is copied into each checked out repository under .ods/umbrella so that it can be accessed from the child repositories (e.g. to make use of Helm value files). TBD: the umbrella directory could also be given as an optional, read-only workspace, but this possibility still has to be explored (e.g. it requires knowledge if the task supports specifying that workspace).

Skipping commits has to be implemented in the webhook interceptor as tasks cannot skip the rest of the pipeline. For this, the interceptor needs to get the last commit message via API from Bitbucket.

• Tasks need to defined in YAML, and the definition is quite verbose (e.g. workspace needs to be repeated). We try to mitigate by requiring as few params as necessary.
• Using Tekton task results is only possible to a limited extend as tasks might be included in different pipelines (see multi-repo pipelines)
• The task / pipeline defintion is quite static, which makes it hard to achieve e.g. different customizations based on branch (such as different flags to provide to Helm). I think this general limitation is an attribute of Tekton which one has to live with ...
• Setting final Bitbucket build status is not implemened yet (this should be doable in a finally task)
• No pass/fail notification yet (this should be doable in a finally task which we need anyway to set the Bitbucket build status ... e.g. it would be nice to have a POST request to a Teams channel ... but how to configure this? A ConfigMap? An env var in the webhook interceptor? config in ods.yml?)
• External triggers (e.g. from Jira) are not implemented in this concept yet. They could just use the same input. The payload could be differentiated based on one "kind"/"event" key. The authentication would require the requester to apply HMAC signing, which would be nice for additional security but will be harder to use ad-hoc.
• when expressions as used above do not actually work like this: they do not skip the current task only, but the whole following branch. There is work in Tekton to make this configurable. We either need to wait for this, or implement the task skipping in the webhook interceptor ourselves.
• We still need to think through and describe how to deal with race conditions. E.g. a multi-repo pipeline might depend on artefacts provided by single-repo pipelines, and this dependency should be easy to understand, and execution of pipelines should be reliable.