This demo will show how to prepare a useful MPL pipeline for a kubernetes jenkins environment and support infrastructure (git service, artifact storage).

We will use minikube, gitea, nexus, jenkins, unit and parallel integration tests, common lifecycle dev -> qa -> prod.

Uses a modified petclinic example: https://github.com/deors/deors-demos-petclinic

• This is just an example of the infrastructure - it doesn't have a proper security enforcement (but could be easily added).
• All the variables we entering manually here are set by default in the preconfiguration - so if you want to change something grep and change the configs first.
• Make sure you have enough resources: I executed the example on the instance with 8vCPU and 16GB of RAM.

We will use kubernetes to simplify the envs separation and use the better jenkins agents.

1. Install kubectl & minikube - there is a number of ways, but for MacOS it's better to use Docker Desktop, for linux and for windows: https://kubernetes.io/docs/tasks/tools/install-minikube/
   • If you using linux - make sure your /tmp directory is mounted with no noexec. That's needed for PVC's that uses /tmp/hostpath-provisioner directory to store the dirs. Otherwise the applications requires execution on their data (git uses exec hooks in the repos) will fail.
2. Setup kubernetes dashboard - it's useful for monitoring and showing what's happening on the cluster. Choose the right version for your minikube (install could be different depends on the version you're going to use): https://github.com/kubernetes/dashboard/releases

   $ KUBE_DB_VERSION=v2.0.0
   
   $ kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/${KUBE_DB_VERSION}/aio/deploy/recommended.yaml
   namespace/kubernetes-dashboard created
   serviceaccount/kubernetes-dashboard created
   service/kubernetes-dashboard created
   secret/kubernetes-dashboard-certs created
   secret/kubernetes-dashboard-csrf created
   secret/kubernetes-dashboard-key-holder created
   configmap/kubernetes-dashboard-settings created
   role.rbac.authorization.k8s.io/kubernetes-dashboard created
   clusterrole.rbac.authorization.k8s.io/kubernetes-dashboard created
   rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
   clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
   deployment.apps/kubernetes-dashboard created
   service/dashboard-metrics-scraper created
   deployment.apps/dashboard-metrics-scraper created
   

3. Apply the service admin-user account to access dashboard:

   $ kubectl apply -f k8s-components/01-dahsboard-access.yaml
   serviceaccount/admin-user created
   clusterrolebinding.rbac.authorization.k8s.io/admin-user created
   

4. Setup the namespaces we will use for our infrastructure and applications:

   $ kubectl apply -f k8s-components/02-namespaces.yaml
   namespace/example-infra created
   namespace/example-dev created
   namespace/example-qa created
   namespace/example-prod created
   

5. Start local proxy to access the kubernetes services:

   $ kubectl proxy
   

6. You can login to the k8s dasboard using token from the next command and url: http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/

   $ kubectl -n kubernetes-dashboard describe secret $(kubectl -n kubernetes-dashboard get secret | grep admin-user | awk '{print $1}')
   <output with token will be printed here>
   

7. Check that the namespaces was created on the dashboard

Ok, now we have a solid base for our system. Next we need to install the components will help us to setup the infrastructure.

This small git server will help us to trigger the jenkins jobs when there some changes will come.

1. Setup simple gitea git server (downloading of the image could take some time)

   $ kubectl apply -f k8s-components/03-gitea-pvc.yaml,k8s-components/03-gitea.yaml
   persistentvolumeclaim/gitea-data-pvc created
   service/git-service created
   statefulset.apps/gitea-app created
   

2. Gitea working not well with proxy, so - port-forwarding:

   $ kubectl -n example-infra port-forward service/gitea-service 3000:web
   Forwarding from 127.0.0.1:3000 -> 3000
   Forwarding from [::1]:3000 -> 3000
   

3. Configure gitea: go to http://localhost:3000/install
   • Select SQLite3 database
   • Go to Server and Third-Party Service Settings and select Enable Local Mode
   • Go to Administrator Account Settings and create "git-admin" account
   • Click Install button

Will store our maven artifacts right after the build.

1. Install nexus3 artifact server (downloading of the image could take some time)

   $ kubectl apply -f k8s-components/04-nexus-pvc.yaml,k8s-components/04-nexus.yaml
   persistentvolumeclaim/nexus-data-pvc created
   service/nexus-service created
   statefulset.apps/nexus-app created
   

2. Run the port forwarding (nexus UI don't like proxy, make sure socat is installed on linux)

   $ kubectl -n example-infra port-forward service/nexus-service 4000:web
   Forwarding from 127.0.0.1:4000 -> 8080
   Forwarding from [::1]:4000 -> 8080
   

3. Login and setup nexus:
   • Go to the nexus web page and click Sign In button: http://localhost:4000/
   • Use dashboard Exec into Pod button or docker exec command to cat the /nexus-data/admin.password file on the container
   • Change admin password
   • Set Enable anonymous access flag
4. Add jenkinsci repository proxy (we need it to build shared libraries):
   • Go to repositories: http://localhost:4000/#admin/repository/repositories
   • Click Create repository button
   • Select "maven2 (proxy)"
   • Set Name "jenkinsci"
   • Set Remote storage to "http://repo.jenkins-ci.org/public/"
   • Click Create repository button at the bottom
   • Go to "maven-public" group: http://localhost:4000/#admin/repository/repositories:maven-public
   • Add "jenkinsci" to Members list
   • Click Save button at the bottom
5. Create new role to handle jenkins access:
   • Go to roles settings: http://localhost:4000/#admin/security/roles
   • Click Create role and select "Nexus role" item
   • Set Role ID and Role name to "jenkins-deploy"
   • Move Privileges available item "nx-repository-view-*-*-*" to Given field
   • Click Create role button to save the new role
6. Add jenkins user to jenkins-deploy role:
   • Go to users settings: http://localhost:4000/#admin/security/users
   • Click on Create local user button
   • Set ID and Password to "jenkins-deploy"
   • Select Status "Active"
   • Move role "jenkins-deploy" to Granted
   • Set the other req fields as you like
   • Click to Create local user button to save the new user

We will use jenkinsBro to simplify the installation:

1. Clone the jenkinsBro repository:

   $ git clone https://github.com/rabits/jenkinsbro.git
   

2. Modify the plugins.txt list with required plugins for our pipelines:
   • config-file-provider - maven global settings
   • kubernetes - cloud to run ondemand agents
   • gogs-webhook - automatically trigger builds from gitea git changes (we will use it to build MPL)
   • pipeline-utility-steps - contains useful parser pipeline steps (for pom files for example)
   • custom-tools-plugin - used for custom tools: kaniko

     $ echo "config-file-provider\nkubernetes\ngogs-webhook\npipeline-utility-steps\ncustom-tools-plugin" >> ./jenkinsbro/plugins.txt
     

3. Build the image of the latest jenkins LTS - it will generate jenkinsbro-master image:

   $ ./jenkinsbro/jenkinsbro_build.sh lts
   

4. Install jenkins to the example-infra namespace (yaml file contains preconfiguration, it's a good idea to read it):

   $ kubectl apply -f k8s-components/05-jenkins-pvc.yaml,k8s-components/05-jenkins.yaml
   persistentvolumeclaim/jenkins-data-pvc created
   service/jenkins-service created
   configmap/jenkinsbro-config created
   statefulset.apps/jenkins-app created
   

We will use slightly advanced kubernetes agent pod - it will be built from 2 containers: jnlp and worker. This is needed for a simple thing - to properly limit the memory consumed by the workload and to make sure agent will not be killed by kubernetes if workload will exceed the memory limit. BTW container will use jnlp agent provided by our jenkins master.

• Build the image of jnlp agent available in the real-demo directory:

  $ docker build -t jenkins-agent-jnlp k8s-build/jenkins-agent-jnlp
  

Ok, we prepared the base infrastructure and now need to fill it with some useful stuff. Since this example is about MPL, we will create the nested shared library with our overrides to build the simple MPL self-build example and store it in nexus to use later.

We will need a mirror of the MPL repository (to make sure internet shutdown will not affect us)

1. Open gitea: http://localhost:3000/
2. Create example organization:
   • Click on + Create menu
   • Select New Organization item
   • Set Organization Name to "example-org"
   • Click Create Organization button
3. Create mirror of the MPL repo in the example-org:
   • Click on + Create menu
   • Select New Migration item
   • Set Owner to "example-org"
   • Set Clone From URL to "https://github.com/griddynamics/mpl"
   • Enable Migration Type flag - we're not going to modify this repo
   • Click Migrate Repository button

We can't use the default MPL build in our infrastructure, so we have to create an additional nested shared library with a number of overrides.

1. Open gitea: http://localhost:3000/
2. Create new repo to store our example project cicd lib:
   • Click on + Create menu
   • Select New Repository item
   • Set Owner to "example-org"
   • Set Repository Name to "example-cicd"
   • Click on Create Repository button
3. Prepare and push the shared lib:
   • Init git repo in the existing shared library dir

     $ git init example-cicd
     Initialized empty Git repository in example-cicd/.git/
     

   • Commit the existing workspace files

     $ git -C example-cicd add .
     $ git -C example-cicd commit . -m 'Init'
     

   • Run the port forwarding to push the data (make sure socat is installed on linux)

     $ kubectl -n example-infra port-forward service/git-service 3000:web
     Forwarding from 127.0.0.1:3000 -> 3000
     Forwarding from [::1]:3000 -> 3000
     

   • Add remote and push the commit

     $ git -C example-cicd remote add origin "http://localhost:3000/example-org/example-cicd.git"
     $ git -C example-cicd push -u origin master
     Handling connection for 3000
     Username for 'http://localhost:3000': git-admin
     Password for 'http://git-admin@localhost:3000':
     Counting objects: 59, done.
     Delta compression using up to 48 threads.
     Compressing objects: 100% (42/42), done.
     Writing objects: 100% (59/59), 13.20 KiB | 1.47 MiB/s, done.
     Total 59 (delta 2), reused 0 (delta 0)
     To http://localhost:3000/example-org/example-cicd.git
      * [new branch]      master -> master
     Branch 'master' set up to track remote branch 'master' from 'origin' by rebasing.
     

Ok, now we're ready to actually setup the pipeline - let's start with minimal MPL build: we need the artifacts stored in nexus to move forward.

1. Run the port forwarding (make sure socat is installed on linux)

   $ kubectl -n example-infra port-forward service/jenkins-service 8089:web
   Forwarding from 127.0.0.1:8089 -> 8080
   Forwarding from [::1]:8089 -> 8080
   

2. Go to Jenkins: http://localhost:8089/
3. Create new job "mpl-master" type "Pipeline"
   • Enable Branch Filter and set "master"
   • In the Pipeline script put the next content:

     @Library('example-cicd') _
     
     ExampleProjectPipeline {
       git = [
         url: 'http://git-service/example-org/mpl',
         branch: 'master',
       ]
       // Disabling not needed stages
       modules.Deploy = null
       modules.Test = null
     }
     

   • Save the job "mpl-master"
4. Set the gogs webhook:
   • Go to gitea mpl repo webhooks settings: http://localhost:3000/example-org/mpl/settings/hooks
   • Click Add Webhook, select "Gogs" item
   • Set Payload URL to "http://jenkins-service/gogs-webhook/?job=mpl-master"
   • Click Add Webhook
5. Go into the created webhook and click Test Delivery button at the bottom
6. Go back to the jenkins "mpl-master" job: http://localhost:8085/job/mpl-master/ and see how build is executing

The example shared library we uploaded contains a number of tests, so we will compile and execute unit tests for each change of the library. Let's create example-cicd-build jenkins job and setup the required hooks:

1. Go to Jenkins: http://localhost:8085/
2. Create new job "example-cicd-build" type "Multibranch Pipeline"
   • Add "Git" to the Branch Sources
   • Set Project Repository "http://git-service/example-org/example-cicd"
   • Click Save button at the bottom - after that multibranch job will check the available branches and start the master build
3. Set the gitea webhook - it will trigger changes build using the git plugin:
   • Go to gitea example-cicd repo webhooks settings: http://localhost:3000/example-org/example-cicd/settings/hooks
   • Click Add Webhook, select "Gogs" item
   • Set Payload URL to "http://jenkins-service/git/notifyCommit?url=http://git-service/example-org/example-cicd"
   • Click Add Webhook

Now we need an application that we could build, deploy, test and move through the release process and environments. We will take a slightly modified petclinic application (as we've used in the petclinic-selenium example).

1. Create new repo to store our petclinic sources:
   • Go into example-org
   • Click Create Repository button
   • Set Repository Name to "petclinic"
   • Click on Create Repository button
2. Prepare and push petclinic sources:
   • Init git repo in the existing petclinic dir

     $ git init petclinic
     Initialized empty Git repository in petclinic/.git/
     

   • Commit the existing workspace files

     $ git -C petclinic add .
     $ git -C petclinic commit . -m 'Init'
     

   • Add remote and push the commit

     $ git -C petclinic remote add origin http://localhost:3000/example-org/petclinic.git
     $ git -C petclinic push -u origin master
     Counting objects: 187, done.
     Delta compression using up to 8 threads.
     Compressing objects: 100% (172/172), done.
     Writing objects: 100% (187/187), 176.47 KiB | 4.52 MiB/s, done.
     Total 187 (delta 40), reused 0 (delta 0)
     remote: Resolving deltas: 100% (40/40), done.
     To http://localhost:3000/example-org/petclinic.git
      * [new branch]      master -> master
     Branch 'master' set up to track remote branch 'master' from 'origin' by rebasing.
     

Ok, we have the application, but to pack it to docker image with k8s - we need a special tool named kaniko. It can build docker image without any access to docker daemon, means we can use it in unprevileged docker container (good for security indeed).

Unfortunately it's oficially distributed only as a docker image, so extracting of the tool will take some time.

1. Create the kaniko container:

   $ docker create --name kaniko gcr.io/kaniko-project/executor:v0.19.0
   Unable to find image 'gcr.io/kaniko-project/executor:v0.19.0' locally
   v0.19.0: Pulling from kaniko-project/executor
   c30f0b4c9053: Pull complete
   ed36162ea2d3: Pull complete
   934aba279703: Pull complete
   958c06b88e30: Pull complete
   9fa803ac1ec6: Pull complete
   03a44a7314d7: Pull complete
   861e678c6f4c: Pull complete
   e60dcc1bf57d: Pull complete
   Digest: sha256:66be3f60f22b571faa82e0aaeb94731217ba0c58ac4a3b062bc84c6d8d545213
   Status: Downloaded newer image for gcr.io/kaniko-project/executor:v0.19.0
   31b512e80ec112293978b6eca12bd7e193c95e6bbcfb8c80f142e8e724321ab5
   

2. Copy the executor to the local fs:

   $ docker cp kaniko:/kaniko/executor /tmp/kaniko
   

3. Delete the docker container:

   $ docker rm kaniko
   

4. Pack the kaniko binary to a zip archive:

   $ zip -9 -j /tmp/kaniko.zip /tmp/kaniko
   

5. Upload the archive to nexus:

   $ curl -u jenkins-deploy:jenkins-deploy --upload-file /tmp/kaniko.zip http://localhost:4000/repository/maven-releases/com.google.GoogleContainerTools/kaniko/0.19.0/kaniko-0.19.0.zip
   

6. Make sure the custom tool kaniko is already created in the global tools of Jenkins - jenkinsbro and our k8s component contains all the required logic and config to setup it.

We will need a private repository to store our built docker images

1. Go to repositories: http://localhost:4000/#admin/repository/repositories
2. Click Create repository button
3. Select "docker (hosted)" item
4. Set Name "docker-registry"
5. Set HTTP flag and enter port "5000"
6. Click Create repository button at the bottom

1. Go to Jenkins: http://localhost:8085/
2. Create new job "petclinic-build" type "Multibranch Pipeline"
   • Add "Git" to the Branch Sources
   • Set Project Repository "http://git-service/example-org/petclinic"
   • Click Save button at the bottom - after that multibranch job will check the available branches and start the master build
3. Set the gitea webhook - it will trigger changes build using the git plugin:
   • Go to gitea petclinic repo webhooks settings: http://localhost:3000/example-org/petclinic/settings/hooks
   • Click Add Webhook, select "Gogs" item
   • Set Payload URL to "http://jenkins-service/git/notifyCommit?url=http://git-service/example-org/petclinic"
   • Click Add Webhook