backups jenkins - juancamilocc/virtual_resources GitHub Wiki

Automated Backup in Jenkins

In this guide, you will learn how to make an automated backup of your Jenkins Server saving it in the cloud storage for different providers like AWS, GCP, Azure and OCI.

The workflow is the following.

Wokflows automated backup Jenkins

To start, we have to install ThinBackup plugin. Go to Manage Jenkins -> Plugins -> Available plugins.

ThinBackup plugin

Now, let's setup the plugin. Go to Manage Jenkins -> System -> ThinBackup Configuration.

Setup ThinBackup plugin

As shows in the previous configuration, we set the path /var/backups where it will save the backups, also set H 22 * * 5 indicating that it will be schedule each friday at 22 hours, and finally set H 22 * * 1-4 for differential backups, keeping a maximum number of copies as 1.

This bash script will do the automated proccess.

#!/bin/bash

BACKUP_DIR="/var/backups"
BUCKET_NAME="backup-jenkins"
CURRENT_DATE=$(TZ="America/Bogota" date +%Y-%m-%d_%H-%M)

# Get the most recent jenkins-server pod
JENKINS_SERVER_POD=$(kubectl -n jenkins get pods --sort-by=.status.startTime | grep jenkins-server | tail -1 | awk '{print $1}')
echo "$JENKINS_SERVER_POD selected pod"

# Verify existence jenkins-server pod
if [ -z "$JENKINS_SERVER_POD" ]; then
    echo "not found jenkins-server pod"
    exit 1
fi

BACKUP_FILE=$(kubectl -n jenkins exec -it "$JENKINS_SERVER_POD" -- bash -c "ls -td $BACKUP_DIR/FULL-* | head -1")

if [ -n "$BACKUP_FILE" ]; then 

    # Compress backup file
    echo "compressing jenkins backup file..."
    kubectl -n jenkins exec "$JENKINS_SERVER_POD" -- tar czvf /tmp/jenkins-backup-$CURRENT_DATE.tar.gz -C "$BACKUP_DIR" .
    echo "jenkins compressed backup file successfully"

    # Copy to cloud container provider
    echo "copying compressed jenkins backup file to cloud container..."
    kubectl cp jenkins/"$JENKINS_SERVER_POD":/tmp/jenkins-backup-$CURRENT_DATE.tar.gz jenkins-backup-$CURRENT_DATE.tar.gz
    echo "compressed jenkins backup file copied successfully"

    echo "deleting compressed backup file in tmp folder..."
    kubectl -n jenkins exec "$JENKINS_SERVER_POD" -- rm -r /tmp/jenkins-backup-$CURRENT_DATE.tar.gz
    echo "compressed backup file deleted successfully"

    # Verify the most recent compressed backup file in oci container
    BACKUP_COMPRESS_FILE=$(ls -t jenkins-backup-*.* | head -1)

    if [ -f "$BACKUP_COMPRESS_FILE" ]; then

        # Uploading compressed jenkins backup file to the bucket
        echo "uploading $BACKUP_COMPRESS_FILE to the bucket $BUCKET_NAME..."
        # ------->>>> Here choose your cloud <<<<---------
        # Using AWS Cloud
        # aws s3 cp "$BACKUP_COMPRESS_FILE" s3://$BUCKET_NAME/
        
        # Using GCP Cloud
        # gsutil cp "$BACKUP_COMPRESS_FILE" gs://$BUCKET_NAME/
        
        # Using Azure Cloud
        # az storage blob upload --account-name <account-name> --container-name $BUCKET_NAME --file "$BACKUP_COMPRESS_FILE" --name "$(basename "$BACKUP_COMPRESS_FILE")"
        
        # Using OCI Cloud 
        # oci os object put --bucket-name $BUCKET_NAME --file "$BACKUP_COMPRESS_FILE" --name "$(basename "$BACKUP_COMPRESS_FILE")"
        echo "compressed jenkins backup file uploaded successfully"
        exit 0

    else

        echo "Not found any compressed jenkins backup file"
    fi

else 

    echo "Not found backup file in the jenkins-server pod"
fi

NOTE: You just have to choose your cloud provider and uncomment the line. Besides as shown in the script this filters for a FULL-* name given by the plugin.

Example backup

We use the following Dockerfile, also here, you must uncomment certain lines depending of your cloud provider.

# Choose your cloud provider

# AWS Cloud
# FROM amazon/aws-cli

# GCP Cloud
# FROM google/cloud-sdk

# Azure Cloud
# FROM chainguard/az:latest-dev

# OCI Cloud
# FROM ghcr.io/oracle/oci-cli:latest 
# ENV OCI_CLI_SUPPRESS_FILE_PERMISSIONS_WARNING=True

USER root

COPY backup-jenkins.sh backup-jenkins.sh

# AWS Cloud
# RUN yum install -y unzip curl
# Azure or GCP Cloud
# RUN apt-get update && apt-get install -y curl
# OCI Cloud
# RUN dnf update -y && dnf install -y curl bash-completion

RUN curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" && \
    chmod +x kubectl && \
    mv kubectl /usr/local/bin/

ENTRYPOINT ["bash", "backup-jenkins.sh"]

Now, we should define the files to deploy in the Kubernetes cluster.

kube-config-cm.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: kube-configmap
  namespace: jenkins
data:
  CONFIG: |
    <here kubeconfig>

In AWS Cloud

For aws config file, as aws-conf.yaml.

NOTE: You should have in count, the aws config file looks like this:

[default]
aws_access_key_id = <tu_access_key_id>
aws_secret_access_key = <tu_secret_access_key>
region = <region>
echo "<aws-config-file>" | base64
# base64 content

aws-config.yaml

apiVersion: v1
kind: Secret
metadata:
  name: aws-config-secret
  namespace: jenkins
data:
  config: <encoded-base64 aws config-file>
type: Opaque

Then, we can set up the cronjob, as follows.

aws-backup-jenkins-cj.yaml

apiVersion: batch/v1
kind: CronJob
metadata:
  name: jenkins-backup-cron
  namespace: jenkins
spec:
  schedule: "0 23 * * 5"
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: jenkins-admin
          containers:
          - name: backup-jenkins
            image: <aws-backup-image> # This is defined in the Dockerfile section
            imagePullPolicy: Always
            volumeMounts:
            - name: aws-config-secret
              mountPath: /root/.aws/config
              subPath: config
            - name: kube-configmap
              mountPath: /root/.kube/
          restartPolicy: OnFailure
          volumes:
          - name: aws-config-secret
            secret:
                secretName: aws-config-secret
                items:
                - key: config
                    path: config 
          - name: kube-configmap
            configMap:
              name: kube-configmap
              items:
                  - key: CONFIG
                    path: config
              defaultMode: 0600

Deploy all resources.

kubectl apply -f aws-conf.yaml
kubectl apply -f kube-config-cm.yaml
kubectl apply -f aws-backup-jenkins-cj.yaml

In GCP Cloud

For gcp config file, as gcp-conf.yaml

NOTE: You should have in count, the gcp config file looks like this:

{
  "type": "service_account",
  "project_id": "PROJECT_ID",
  "private_key_id": "KEY_ID",
  "private_key": "-----BEGIN PRIVATE KEY-----\nPRIVATE_KEY\n-----END PRIVATE KEY-----\n",
  "client_email": "SERVICE_ACCOUNT_EMAIL",
  "client_id": "CLIENT_ID",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://accounts.google.com/o/oauth2/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/SERVICE_ACCOUNT_EMAIL"
}
echo "<gcp-config-file>" | base64
# base64 content

gcp-config.yaml

apiVersion: v1
kind: Secret
metadata:
  name: gcp-config-secret
  namespace: jenkins
data:
  service-account.json: <encoded-base64 gcp-config-file>
type: Opaque

Then, we can set up the cronjob, as follows.

gcp-backup-jenkins-cj.yaml

apiVersion: batch/v1
kind: CronJob
metadata:
  name: jenkins-backup-cron
  namespace: jenkins
spec:
  schedule: "0 23 * * 5"
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: jenkins-admin
          containers:
          - name: backup-jenkins
            image: <gcp-backup-image> # This is defined in the Dockerfile section
            imagePullPolicy: Always
            volumeMounts:
            - name: gcp-config-secret
              mountPath: /root/.config/gcloud/application_default_credentials.json
              subPath: service-account.json
            - name: kube-configmap
              mountPath: /root/.kube/
          restartPolicy: OnFailure
          volumes:
          - name: gcp-config-secret
            secret:
              secretName: gcp-config-secret
              items:
                - key: service-account.json
                  path: service-account.json
          - name: kube-configmap
            configMap:
              name: kube-configmap
              items:
                - key: CONFIG
                  path: config
              defaultMode: 0600

Deploy all resources.

kubectl apply -f gcp-conf.yaml
kubectl apply -f kube-config-cm.yaml
kubectl apply -f gcp-backup-jenkins-cj.yaml

In Azure Cloud

For azure config file, as az-conf.yaml

NOTE: You should have in count, the azure config file looks like this:

{
  "clientId": "CLIENT_ID",
  "clientSecret": "CLIENT_SECRET",
  "tenantId": "TENANT_ID",
  "subscriptionId": "SUBSCRIPTION_ID"
}
echo "<azure-config-file>" | base64
# base64 content

az-config.yaml

apiVersion: v1
kind: Secret
metadata:
  name: azure-config-secret
  namespace: jenkins
data:
  azure-config.json: <encoded-base64 azure-config-file>
type: Opaque

Then, we can set up the cronjob, as follows.

az-backup-jenkins-cj.yaml

apiVersion: batch/v1
kind: CronJob
metadata:
  name: jenkins-backup-cron
  namespace: jenkins
spec:
  schedule: "0 23 * * 5"
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: jenkins-admin
          containers:
          - name: backup-jenkins
            image: <az-backup-image> # This is defined in the Dockerfile section
            imagePullPolicy: Always
            volumeMounts:
            - name: azure-config-secret
              mountPath: /root/.azure/azure.json
              subPath: azure-config.json
            - name: kube-configmap
              mountPath: /root/.kube/
          restartPolicy: OnFailure
          volumes:
          - name: azure-config-secret
            secret:
              secretName: azure-config-secret
              items:
                - key: azure-config.json
                  path: azure-config.json
          - name: kube-configmap
            configMap:
              name: kube-configmap
              items:
                - key: CONFIG
                  path: config
              defaultMode: 0600

Deploy all resources.

kubectl apply -f az-conf.yaml
kubectl apply -f kube-config-cm.yaml
kubectl apply -f az-backup-jenkins-cj.yaml

In OCI Cloud

For oci config file, as oci-conf.yaml.

apiVersion: v1
kind: ConfigMap
metadata:
  name: oci-conf
  namespace: jenkins
data:
  oci_config: |-
    [DEFAULT]
    user=<userid>
    fingerprint=<fingerprint>
    key_file=/root/.oci/oci_api_key.pem
    tenancy=<tenancy>
    region=<region>

For oci key-secret, as oci-key-secret.yam.

apiVersion: v1
kind: Secret
metadata:
  name: oci-key
  namespace: jenkins
data:
  oci_api_key.pem: <base64-encoded key-secret>
type: Opaque

Then, we can set up the cronjob, as follows.

oci-backup-jenkins-cj.yaml

apiVersion: batch/v1
kind: CronJob
metadata:
  name: jenkins-backup-cron
  namespace: jenkins
spec:
  schedule: "0 23 * * 5"
  jobTemplate:
    spec:
      template:
        spec:
          priorityClassName: workflow-controller
          serviceAccountName: jenkins-admin
          containers:
          - name: backup-jenkins
            image: <oci-backup-image> # This is defined in the Dockerfile section
            imagePullPolicy: Always
            volumeMounts:
            - name: oci-key
              mountPath: /root/.oci/oci_api_key.pem
              subPath: oci_api_key.pem
            - name: oci-conf
              mountPath: /root/.oci/config
              subPath: config
            - name: kube-configmap
              mountPath: /root/.kube/
          restartPolicy: OnFailure
          volumes:
          - name: oci-key
            secret:
              secretName: oci-key
              items:
                  - key: oci_api_key.pem
                    path: oci_api_key.pem        
              defaultMode: 256
          - name: oci-conf
            configMap:
              name: oci-conf
              items:
                  - key: oci_config
                    path: config  
          - name: kube-configmap
            configMap:
              name: kube-configmap
              items:
                  - key: CONFIG
                    path: config
              defaultMode: 0600

Deploy all resources.

kubectl apply -f oci-conf.yaml
kubectl apply -f oci-key-secret.yaml
kubectl apply -f kube-config-cm.yaml
kubectl apply -f oci-backup-jenkins-cj.yaml

NOTE: In case to lose all information about your Jenkins Server for any reason, you just have to go to download the backup file from your bucket, install again the plugin ThinBackup, go to Manage Jenkins -> ThinBackup -> Restore and upload the uncompressed backup file.

Restore using backup

Conclusions

Automating backups significantly enhances resilience against disasters and data loss. It also provides speed and ease when reconfiguring all resources on the Jenkins server.

⚠️ **GitHub.com Fallback** ⚠️