Comparison with Jenkins - michaelsauter/ods-pipeline GitHub Wiki

ODS 3.x and earlier uses Jenkins for CI/CD functionality. In general, Tekton pipelines can be seen as an alternative to Jenkins as they can be used to achieve the same things. However, there are some notable differences:

  • Jenkins uses a long-running service (the "Jenkins master"), whereas Tekton does not have a server component by itself. OpenShift Pipelines offers a basic UI to view pipeline runs and their logs. However, several functionalities like Jenkins job attachments are not available in OpenShift Pipelines. One advantage of not having a server component is the reduction of memory usage, which is typically around 5GB per Jenkins instance.
  • Jenkins can be extended with plugins. This is not possible with OpenShift Pipelines. Some functionality provided by plugins can be covered by Tekton tasks, which make use of images providing certain tooling.
  • To avoid repeating common pipeline tasks, ODS shipped with a "Jenkins shared library". This library defines several stages which can be called from the application's Jenkinsfile. The Tekton tasks provided by ODS can be seen as a replacement for the shard library concept. The ods.yml file defining the tasks can be seen as a replacement for the Jenkinsfile.
  • Jenkins allows to script pipelines with Groovy. This opens a lot of possibilites and makes Jenkins very flexible. However, this flexibility comes at a cost as well: it introduces additional complexity and increases the likelihood of bugs. Further, to support resuming pipelines, Jenkins applies so called CPS transformation to the Groovy source code which can have very surprising behaviour and requires a lot of detailed knowledge about the inner workings of Jenkins.
  • The Jenkins master/agent concept led to a brittle master/agent dance back-and-forth in the shared library orchestration pipeline because some things need to run on the master (e.g. executing sub-pipelines), and other things need to run on the agent (e.g. external cluster deploy).

Using Tekton Pipelines under the hood allows us to benefit from the following properties:

  • Task definitions are plain Tekton tasks - any example from the internet can be copy/pasted. It is not necessary to use a base ODS image etc.
  • In ODS 3.x, each repository had exactly one "component type", and the release manager component (multi-repo orchestration) dealt with each type differently. This concept is not needed with the new pipeline approach, allowing users to have one or more "components" within one repository - even everything in one monorepo.
  • Timeout, retries, sidecars per task are built-in to Tekton.
  • Users can create their own tasks in their namespace, and reuse those tasks across multiple repos / pipelines easily.
  • Pipeline runs have a copy of the exact pipeline spec and the tasks specs so even if the tasks are modified over time, it is possible to know exactly what happened in a certain pipeline run.
  • Tekton tasks are based on images which are easier to test than Jenkins stages which depend on way more.
  • Our Jenkins approach was restricted to one agent pod per pipeline. This made it tricky for monorepos, as they often need multiple toolchains. This is very easy to achieve with multiple tasks (which can run in parallel even when mounting the same workspace as they are scheduled on the same node).

Relationship to the Release Manager of ODS 3.x

The "release manager" concept in ODS 3.x consists of three main items:

  • The release manager quickstarter (mainly providing a YAML config file)
  • The orchestration pipeline (part of the shared library)
  • A document generation service (using templates stored in Bitbucket)

Worth noting is that the release manager is able to generate GAMP documentation right now, using information stored in Jira.

The concept above has been developed with this use-case in mind, but it might be a longer process to fully support it. On a high level, here are some notes on the two approaches:

  • The "umbrella" repository is basically exactly what the "release manager" repository is right now (likely with some modifications to the config structure).
  • The orchestration pipeline consists of several phases, and this concept is reflected in the pipeline phases in ods.yml.
  • The orchestration pipeline supports a hook mechansim, allowing to execute code before/after phases, and before/after repos. This is used to generate the documentation in the phase it most closely belongs to (making use of the docgen service). The new concept above does not have a hook mechanism, and part of its power is allowing to run any ("non-ODS") task/image which would be incompatible with a hook mechanism. Therefore, the envisioned way to generate all documentation is to do this at the very end, in a task referenced from the umbrella repository. A nice side-effect of this is that all the docgen logic can be put into the image powering that task, eliminating the need to run and maintain a docgen service in each *-cd namespace. This will also reduces the "release manager" to just one "object", which is the "umbrella" repository - therefore we can just provision it together with a new project, making adoption and integration even better (removing the need for a quickstarter altogether).
  • For the 3.x ods and ods-service component types, the orchestration pipeline just executed each Jenkinsfile in the build phase. This has several drawbacks: e.g. it runs build AND deploy in the build phase, and for promotion to QA and PROD, the Jenkinsfile isn't executed at all, which makes it impossible for each repository to customize the deployment according to the need. The new concept does not have this limitation as the multi-repo pipeline is a collection of tasks extracted from the ods.yml in each repository.
  • The orchestration pipeline relies heavily on information in Jira, which is downloaded at the beginning of each pipeline, and then stored in the Project class. The new concept has no specific, built-in ties to Jira. However, one can build a Tekton task, referenced from the "umbrella" repository, that downloads the Jira data and uploads it to Nexus, making it available later on to the document generation task.
  • The orchestration pipeline has quite a few features right now which are left out in the new concept on purpose. Before adding them back in (even if only to an optional task), we should have a close look if providing and maintaining those features is worth the added complexity. Here is a collection of things which are not covered intentionally:
    • Export of live OpenShift resources into templates (doesn't work well, and shouldn't be done anyway).
    • Any Git branch / tag creation or committing of files (enables to have the stable Git commit SHA on which some of the concept above relies).
    • No merge-back from release branch into main branch (not really necessary anymore as no commits are made, and Bitbucket offers a merge-back functionality in itself for pull requests).
  • The 3.x orchestration pipeline has built-in knowledge how to generate GxP documentation (which documents to generate, and which content to assemble for them). This couples the orchestration aspect with a certain documentation process. The new concept decouples that. Multi-repo pipelines are completely independent from any documentation concern, and repository pipelines simply store generic artefacts in Nexus that do not correlate to a specific process, but to the software (e.g. xUnit files for test reports). All documentation aspects are handled solely in the docgen task, allowing to have different such tasks (e.g. one for SaMD and one for GxP), that can differ in which types of documents they generate, but also in which content they assemble for those documents.