Recipes - michaelsauter/ods-pipeline GitHub Wiki

Building components in monorepos in parallel

Assuming every component is stored in a subfolder in your repo, you can easily build components in parallel, and avoid rebuilding components that do not need to rebuild in subsequent commits.

Assume your repository contains two folders backend and frontend, your tasks may look like this:

phases:

  build:

  - name: backend-build-go
    taskRef:
      kind: ClusterTask
      name: ods-build-go
    workspaces:
    - name: source
      workspace: shared-workspace
      subPath: backend

  - name: backend-build-image
    taskRef:
      kind: ClusterTask
      name: ods-build-image
    runAfter:
    - backend-build-go
    params:
    - name: image-stream
      value: backend
    workspaces:
    - name: source
      workspace: shared-workspace
      subPath: backend

  - name: frontend-build-nodejs
    taskRef:
      kind: ClusterTask
      name: ods-build-nodejs
    workspaces:
    - name: source
      workspace: shared-workspace
      subPath: frontend

  - name: frontend-build-image
    taskRef:
      kind: ClusterTask
      name: ods-build-image
    runAfter:
    - frontend-build-nodejs
    params:
    - name: image-stream
      value: frontend
    workspaces:
    - name: source
      workspace: shared-workspace
      subPath: frontend

  deploy:

  - name: backend-deploy
    taskRef:
      kind: ClusterTask
      name: ods-deploy-helm
    params:
    - name: release-name
      value: backend
    workspaces:
    - name: source
      workspace: shared-workspace
      subPath: backend

  - name: frontend-deploy
    taskRef:
      kind: ClusterTask
      name: ods-deploy-helm
    params:
    - name: release-name
      value: frontend
    workspaces:
    - name: source
      workspace: shared-workspace
      subPath: frontend

As there are no dependencies defined between the frontend and backend "branches", OpenShift will automatically schedule them in parallel. This works even though they mount the same volume (backing up the workspace), because Tekton schedules all tasks of a pipeline on the same node by default (making use of an affinity assistant).

Important to note is that you need to explicitly set the image-stream and release-name parameters, because the default would be derived from the repository name, which is not a good fit in this case.

Further, notice the subPath field in the workspace configuration. This ensures that the volume is mounted at this subpath, and therefore the working directory will be set to that folder. As a consequence, the tasks have the sources in the expected location. And there is another benefit from this: the tasks will inspect the Git commit SHA of the subpath, which might differ from the overall repository Git commit SHA. If the Git commit SHA has not changed since a previous build, the task can discover previous artefacts (such as the container image) and avoid to rebuild.

TODO: When document generation is required, that task will then have to deal with the situation that content must be collected from a variety of Git commit SHAs. The concept for this is not done yet but it should be possible.

Deploying to destinations without proper access

It might be desirable that developers have very limited access to the deploy destination. For example, view permissions might be OK but access to pods or secrets should not be allowed.

As deployment requires a serviceaccount with at least edit permissions in the destination namespace, the namespace in which the deploying pipeline runs must necessarily be one where the developers also do no have access to. Therefore, a second *-cd namespace must be created with limited access.

The Bitbucket repos could be configured with an additional webhook pointing to the second *-cd namespace. The pipeline in that namespace should be able to pull images from the first *-cd namespace. That way images do not need to be rebuild. Any Kubernetes Secret resources need to be encrypted for a private key which is only accessible from within the second *-cd namespace.