Taints and Tolerations in MinIO - cniackz/public GitHub Wiki

Objective:

To show a working example on how to set Taints and Tolerations in MinIO Tenant

Documentation:

• https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/

Steps:

1. Have a cluster ready

2. Taint the nodes:

kubectl taint nodes kind-worker k8s-app=minio:NoSchedule
kubectl taint nodes kind-worker2 k8s-app=minio:NoSchedule
kubectl taint nodes kind-worker3 k8s-app=minio:NoSchedule
kubectl taint nodes kind-worker4 k8s-app=minio:NoSchedule

Expected result:

node/kind-worker tainted
node/kind-worker2 tainted
node/kind-worker3 tainted
node/kind-worker4 tainted

3. Deploy a Tenant without any Toleration, pods will be in pending state due to missing toleration in tenant spec:

$ k get pods -n default
NAME             READY   STATUS    RESTARTS   AGE
celis-pool-0-0   0/2     Pending   0          20s
celis-pool-0-1   0/2     Pending   0          20s
celis-pool-0-2   0/2     Pending   0          20s
celis-pool-0-3   0/2     Pending   0          19s

4. Add the toleration in the tenant spec:

$ k get tenant -n default
NAME    STATE                        AGE
celis   Provisioning initial users   50s

$ k edit tenant celis -n default

apiVersion: minio.min.io/v2
kind: Tenant
spec:
  pools:
  - name: pool-0
    resources: {}
    runtimeClassName: ""
    servers: 4
    tolerations:
    - effect: NoSchedule
      key: k8s-app
      operator: Equal
      value: minio

5. After some time all MinIO pods will be running because they have the toleration now:

$ k get pods -n default
NAME             READY   STATUS    RESTARTS   AGE
celis-pool-0-0   2/2     Running   0          4m54s
celis-pool-0-1   2/2     Running   0          5m13s
celis-pool-0-2   2/2     Running   0          5m42s
celis-pool-0-3   2/2     Running   0          6m9s