Specific instructions to run with Kubernetes cluster - PanDAWMS/panda-harvester GitHub Wiki

Kubernetes Plugin

Handle CVMFS

A few possible ways:

  • Install CVMFS on every Kubernetes node and mount.
  • At CERN, use cvmfs-csi as a k8s pvc and sc to mount in container.

Harvester server

Installation procedures

After finished Installation and Configuration, please install kubernetes python client:

pip install kubernetes

Setup and system configuration files

YAML file

Using the Job kind to prevent Pod from occupying resources after it completed.
The following list shows parameters need to be modified:

Name Description
metadata.name Job name
spec.template.spec.containers.name Container name
spec.template.spec.containers.image `atlasadc/atlas-grid-centos6` or `atlasadc/atlas-grid-slc6`

The command content will be executed to setup container environment and then pull pilot to run.
For example,

apiVersion: batch/v1
kind: Job
metadata:
  name: atlasadc-job
spec:
  template:
    spec:
      containers:
        - name: atlas-grid-centos6
          image: atlasadc/atlas-grid-centos6
          volumeMounts:
            - name: cvmfs
              mountPath: /cvmfs
          imagePullPolicy: IfNotPresent
          command:
            - "sh"
            - "-c"
            - >
              curl -4 https://bootstrap.pypa.io/2.6/get-pip.py -o /root/get-pip.py;
              if [ -s /root/get-pip.py ]; then
                 python /root/get-pip.py;
                 pip install requests subprocess32;
              fi;
              echo -e "export computingSite=$computingSite\nexport pandaQueueName=$pandaQueueName\nexport proxyContent='$proxyContent'\nexport workerID=$workerID\nexport logs_frontend_w=$logs_frontend_w\nexport logs_frontend_r=$logs_frontend_r\n" > /etc/profile.d/job-setup.sh;
              groupadd -g 1308 atlasprd;
              useradd -u 41000 -g 1308 atlasprd;
              su - atlasprd -c "cd /home/atlasprd; curl -4 https://raw.githubusercontent.com/mightqxc/panda-harvester/k8s/pandaharvester/harvestercloud/k8s_startup_script.py -o /home/atlasprd/k8s_startup_script.py; python /home/atlasprd/k8s_startup_script.py";
          securityContext:
            allowPrivilegeEscalation: false
      restartPolicy: Never
      volumes:
        - name: cvmfs
          hostPath:
            path: /cvmfs
          type: Directory

Set more than one container in YAML also works, but note that one worker is mapped to one container.
The second container (or more) setup in YAML will be ignored.

If you run job on the Kubernetes cluster at CERN, please refer to CERN cloud document to setup CVMFS.
See: http://clouddocs.web.cern.ch/clouddocs/containers/tutorials/cvmfs.html
An example of setting CVMFS on nodes,

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: csi-cvmfs-atlas
provisioner: csi-cvmfsplugin
parameters:
  repository: atlas.cern.ch
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: csi-cvmfs-sft
provisioner: csi-cvmfsplugin
parameters:
  repository: sft.cern.ch
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: csi-cvmfs-grid
provisioner: csi-cvmfsplugin
parameters:
  repository: grid.cern.ch
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: csi-cvmfs-atlas-pvc
spec:
  accessModes:
  - ReadOnlyMany
  resources:
    requests:
      storage: 1Gi
  storageClassName: csi-cvmfs-atlas
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: csi-cvmfs-sft-pvc
spec:
  accessModes:
  - ReadOnlyMany
  resources:
    requests:
      storage: 1Gi
  storageClassName: csi-cvmfs-sft
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: csi-cvmfs-grid-pvc
spec:
  accessModes:
  - ReadOnlyMany
  resources:
    requests:
      storage: 1Gi
  storageClassName: csi-cvmfs-grid

and job YAML,

apiVersion: batch/v1
kind: Job
metadata:
  name: atlasadc-job
spec:
  template:
    spec:
      containers:
        - name: atlas-grid-centos6
          image: atlasadc/atlas-grid-centos6
          volumeMounts:
            - name: atlas
              mountPath: /cvmfs/atlas.cern.ch
            - name: sft
              mountPath: /cvmfs/sft.cern.ch
            - name: grid
              mountPath: /cvmfs/grid.cern.ch
          imagePullPolicy: IfNotPresent
          command:
            - "sh"
            - "-c"
            - >
              curl -4 https://bootstrap.pypa.io/2.6/get-pip.py -o /root/get-pip.py;
              if [ -s /root/get-pip.py ]; then
                 python /root/get-pip.py;
                 pip install requests subprocess32;
              fi;
              echo -e "export computingSite=$computingSite\nexport pandaQueueName=$pandaQueueName\nexport proxyContent='$proxyContent'\nexport workerID=$workerID\nexport logs_frontend_w=$logs_frontend_w\nexport logs_frontend_r=$logs_frontend_r\n" > /etc/profile.d/job-setup.sh;
              groupadd -g 1308 atlasprd;
              useradd -u 41000 -g 1308 atlasprd;
              su - atlasprd -c "cd /home/atlasprd; curl -4 https://raw.githubusercontent.com/mightqxc/panda-harvester/k8s/pandaharvester/harvestercloud/k8s_startup_script.py -o /home/atlasprd/k8s_startup_script.py; python /home/atlasprd/k8s_startup_script.py";
          securityContext:
            allowPrivilegeEscalation: false
      restartPolicy: Never
      volumes:
        - name: atlas
          persistentVolumeClaim:
            claimName: csi-cvmfs-atlas-pvc
            readOnly: true
        - name: sft
          persistentVolumeClaim:
            claimName: csi-cvmfs-sft-pvc
            readOnly: true
        - name: grid
          persistentVolumeClaim:
            claimName: csi-cvmfs-grid-pvc
            readOnly: true
  backoffLimit: 0

Queue Config

Before start Kubernetes plugin, the module and class name should be set in $PANDA_HOME/etc/panda/panda_queueconfig.json.
Also some parameters need to be adjusted:

Name Description
proxySecretPath Path of the proxy file inside container. Can work with the k8s secret managing proxy
x509UserProxy Proxy file path on Harvester node to pass to container. Only works if proxySecretPath NOT set
cpuAdjustRatio Set ratio to adjust resource of CPU before pod creating (default is 100)
memoryAdjustRatio Set ratio to adjust resource of memory before pod creating (default is 100)
k8s_yaml_file YAML file path which creates a kubernetes job
k8s_config_file Configuration file path for Kubernetes client authentication
k8s_namespace If you want to distinguish multiple teams or projects on cluster. See:https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/#when-to-use-multiple-namespaces

For example,

"ANALY_TAIWAN_TEST": {
…...
                "submitter": {
                        "name":"K8sSubmitter",
                        "module":"pandaharvester.harvestersubmitter.k8s_submitter",
                        "x509UserProxy": "/root/atlas-production.proxy"
                        "cpuAdjustRatio": 90,
                        "memoryAdjustRatio": 100
…...
                "monitor": {
                        "name":"K8sMonitor",
                        "module":"pandaharvester.harvestermonitor.k8s_monitor"
                },
                "sweeper": {
                        "name": "K8sSweeper",
                        "module": "pandaharvester.harvestersweeper.k8s_sweeper"
                },
                "common": {
                       "k8s_yaml_file": "/home/harvesteruser/atlas_job.yaml",
                       "k8s_config_file": "/home/harvesteruser/.kube/config",
                       "k8s_namespace": "default"
                }
        },

Special Cases:

Utilize K8S secret credential manager for proxy files

Now Harvester has a credential manager plugin k8s_secret_cred_manager to create/update a k8s secret object.

One can thus export proxy files into containers via k8s secret, and configure harvester to use k8s_secret_cred_manager to update the proxy periodically.

Configuration of k8s_secret_credmanager

One needs to create a configuration file in JSON format for k8s_secret_credmanager.

Important keys are k8s_namespace, k8s_config_file, proxy_files.

Note the proxy files listed in proxy_files must be updated periodically by harvester no_voms_credmanager or in other ways. Thus, k8s_secret_credmanager can update the newest proxy files to k8s secret.

Example of config json file of k8s_secret_credmanager (/opt/harvester_k8s/k8s_secret_cred_manager_config.json in the example above):

{
  "k8s_namespace": "",
  "k8s_config_file": "/opt/harvester_k8s/kubeconf",
  "proxy_files": ["/data/atlpan/atlas.prod.proxy", "/data/atlpan/atlas.pilot.proxy"]
}

Harvester configuration

In panda_harvester.cfg, one needs to add lines of k8s_secret_credmanager in credmanager block.

Here moduleName is pandaharvester.harvestercredmanager.k8s_secret_cred_manager and className is K8sSecretCredManager.

Put the path of configuration file of k8s_secret_credmanager mentioned above at certFile.

Many other attributes are useless for k8s_secret_credmanager.

Example of credmanager block in panda_harvester.cfg:

[credmanager]

# module name
moduleName =
 ...
 pandaharvester.harvestercredmanager.k8s_secret_cred_manager

# class name
className =
 ...
 K8sSecretCredManager

# original certificate file to generate new short-lived certificate
certFile =
 ...
 /opt/harvester_k8s/k8s_secret_cred_manager_config.json

# the name of short-lived certificate
outCertFile =
 ...
 useless_string

# voms
voms =
 ...
 useless_string

# sleep interval in sec
sleepTime = 1800

K8S job yaml

When using k8s_secret_cred_manager, a k8s secret object with name proxy-secret will be created.

In the yaml file, one needs to add a volume of secret with secretName: proxy-secret , and mount it in the container, with a proper mountPath.

Example of yaml file of k8s job:

apiVersion: batch/v1
kind: Job
metadata:
  name: atlasadc-job
spec:
  template:
    spec:
      containers:
        - name: atlas-grid-slc6
          image: atlasadc/atlas-grid-slc6
          volumeMounts:
            - name: atlas
              mountPath: /cvmfs/atlas.cern.ch
            - name: atlas-condb
              mountPath: /cvmfs/atlas-condb.cern.ch
            - name: atlas-nightlies
              mountPath: /cvmfs/atlas-nightlies.cern.ch
            - name: sft
              mountPath: /cvmfs/sft.cern.ch
            - name: grid
              mountPath: /cvmfs/grid.cern.ch
            - name: proxy-secret
              mountPath: /proxy
          imagePullPolicy: IfNotPresent
          #resources:
          #  requests:
          #    memory: "1.5Gi"
          command:
            - "sh"
            - "-c"
            - >
              curl -4 https://bootstrap.pypa.io/2.6/get-pip.py -o /root/get-pip.py;
              if [ -s /root/get-pip.py ]; then
                 python /root/get-pip.py;
                 pip install requests subprocess32;
              fi;
              echo -e "export computingSite=$computingSite\nexport pandaQueueName=$pandaQueueName\nexport proxySecretPath=$proxySecretPath\nexport proxyContent='$proxyContent'\nexport workerID=$workerID\nexport logs_frontend_w=$logs_frontend_w\nexport logs_frontend_r=$logs_frontend_r\nexport PANDA_JSID=$PANDA_JSID\nexport resourceType=$resourceType\n" > /etc/profile.d/job-setup.sh;
              groupadd -g 1308 atlasprd;
              useradd -u 41000 -g 1308 atlasprd;
              su - atlasprd -c "cd /home/atlasprd; curl -4 https://raw.githubusercontent.com/PanDAWMS/panda-harvester/flin/pandaharvester/harvestercloud/k8s_startup_script.py -o /home/atlasprd/k8s_startup_script.py; python /home/atlasprd/k8s_startup_script.py";
          securityContext:
            allowPrivilegeEscalation: false

      restartPolicy: Never
      volumes:
        - name: atlas
          persistentVolumeClaim:
            claimName: csi-cvmfs-atlas-pvc
            readOnly: true
        - name: atlas-condb
          persistentVolumeClaim:
            claimName: csi-cvmfs-atlas-condb-pvc
            readOnly: true
        - name: atlas-nightlies
          persistentVolumeClaim:
            claimName: csi-cvmfs-atlas-nightlies-pvc
            readOnly: true
        - name: sft
          persistentVolumeClaim:
            claimName: csi-cvmfs-sft-pvc
            readOnly: true
        - name: grid
          persistentVolumeClaim:
            claimName: csi-cvmfs-grid-pvc
            readOnly: true
        - name: proxy-secret
          secret:
              secretName: proxy-secret
  backoffLimit: 0

Queue configuration

In queue configuration submitter block, one needs to add the line of proxySecretPath. Note the value of proxySecretPath must be the proxy file path inside the container, basically corresponding to the mountPath setup in yaml and proxy_files defined in configuration json of k8s_secret_cred_manager.

Example of queue configuration json file:

"CERN-EXTENSION_K8S_HARVESTER": {
                "queueStatus": "online",
                "prodSourceLabel": "managed",
                "nQueueLimitWorker": 100,
                "maxWorkers": 1000,
                "maxNewWorkersPerCycle": 30,
                "runMode":"slave",
                "mapType": "NoJob",
                "truePilot": true,
                "preparator": {
                        "name": "DummyPreparator",
                        "module": "pandaharvester.harvesterpreparator.dummy_preparator"
                },
                "submitter": {
                        "name":"K8sSubmitter",
                        "module":"pandaharvester.harvestersubmitter.k8s_submitter",
                        "proxySecretPath":"/proxy/atlas.prod.proxy",
                        "x509UserProxy": "/data/atlpan/x509up_u25606_production"
                },
                "workerMaker": {
                        "name": "SimpleWorkerMaker",
                        "module": "pandaharvester.harvesterworkermaker.simple_worker_maker"
                },
                "messenger": {
                        "name": "SharedFileMessenger",
                        "module": "pandaharvester.harvestermessenger.shared_file_messenger",
                        "accessPoint": "/data/atlpan/harvester_wdirs/${harvesterID}/${_workerID_3.2}/${_workerID_1.0}/${workerID}"
                },
                "stager": {
                        "name": "DummyStager",
                        "module": "pandaharvester.harvesterstager.dummy_stager"
                },
                "monitor": {
                        "name":"K8sMonitor",
                        "module":"pandaharvester.harvestermonitor.k8s_monitor"
                },
                "sweeper": {
                        "name": "K8sSweeper",
                        "module": "pandaharvester.harvestersweeper.k8s_sweeper"
                },
                "common": {
                       "k8s_yaml_file": "/opt/harvester_k8s/k8s_atlas_job_prod_secret.yaml",
                       "k8s_config_file": "/opt/harvester_k8s/kubeconf",
                       "k8s_namespace": ""
                }
    }

K8s scheduling: packing nodes rather than spreading

K8s default scheduling spreads the pods across the nodes with a Round Robin algorithm. This can cause single core pods to spread across all nodes and preventing multi core pods to be scheduled. You can define a custom scheduling policy. Here is the example that worked for us:

  • On the master node define the policy file, including priority stratagy "{"name" : "MostRequestedPriority", "weight" : 1}" at /etc/kubernetes/scheduler-policy.json
{
  "kind" : "Policy",
  "apiVersion" : "v1",
  "predicates" : [
    {"name" : "GeneralPredicates"},
    {"name" : "MatchInterPodAffinity"},
    {"name" : "NoDiskConflict"},
    {"name" : "NoVolumeZoneConflict"},
    {"name" : "PodToleratesNodeTaints"}
    ],
  "priorities" : [
    {"name" : "MostRequestedPriority", "weight" : 1},
    {"name" : "InterPodAffinityPriority", "weight" : 2}
    ]
  }
  • In /etc/kubernetes/scheduler refer to the policy config file in KUBE_SCHEDULER_ARGS:
  KUBE_SCHEDULER_ARGS="--leader-elect=true --policy-config-file /etc/kubernetes/scheduler-policy.json"
  • Then restart scheduler to make the changes take effect:
  $ systemctl restart kube-scheduler.service

Authored by Mandy Yang, FaHui Lin

⚠️ **GitHub.com Fallback** ⚠️