KES cert manager - cniackz/public GitHub Wiki

Tested:

• Thu Jan 26, 2023 @ 12:14 am Toronto Time

Objective:

Add KES to a Tenant that is using cert-manager for the TLS.

Inspired from:

• https://github.com/cniackz/public/wiki/How-to-install-MinIO-Tenant-using-cert-manager-in-k8s
• https://github.com/cniackz/public/wiki/Tenant-TLS-plus-KES-TLS-all-with-cert-manager-together

Steps:

1. First get tenant with cert-manager

createcluster

installoperator

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.11.0/cert-manager.yaml

sleep 120 # wait 2 minutes
k apply -k ~/operator/examples/kustomization/tenant-certmanager

2. Add kes

subl ~/operator/examples/vault/deployment.yaml

• Add vault in tenant-certmanager namespace

---
apiVersion: v1
kind: Service
metadata:
  name: vault
  namespace: tenant-certmanager
  labels:
    name: vault
spec:
  ports:
    - port: 8200
      name: http
  selector:
    app: vault
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vault
  namespace: tenant-certmanager
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vault
  template:
    metadata:
      labels:
        app: vault
    spec:
      containers:
        - name: vault
          image: vault:latest
          imagePullPolicy: "IfNotPresent"
          env:
            - name: SECRET_SHARES
              value: "5"
            - name: SECRET_THRESHOLD
              value: "3"
            - name: SELF_SIGNED_CERT
              value: "true"
            - name: TOTAL_INIT_RETRIES
              value: "5"
          ports:
            - containerPort: 8200
              name: http
          securityContext:
            capabilities:
              add:
                - IPC_LOCK

kubectl apply -f ~/operator/examples/vault/deployment.yaml
kubectl wait --namespace tenant-certmanager --for=condition=ready pod --selector=app=vault --timeout=120s

• Configure Vault:

VAULT_ROOT_TOKEN=$(kubectl logs -l app=vault -n tenant-certmanager | grep "Root Token: " | sed -e "s/Root Token: //g");

kubectl exec $(kubectl get pods -l app=vault -n tenant-certmanager  | grep -v NAME | awk '{print $1}') -n tenant-certmanager -- sh -c 'VAULT_TOKEN='$VAULT_ROOT_TOKEN' VAULT_ADDR="http://127.0.0.1:8200" vault auth enable approle'

kubectl exec $(kubectl get pods -l app=vault -n tenant-certmanager | grep -v NAME | awk '{print $1}') -n tenant-certmanager -- sh -c 'VAULT_TOKEN='$VAULT_ROOT_TOKEN' VAULT_ADDR="http://127.0.0.1:8200" vault secrets enable kv'

kubectl cp ~/operator/examples/vault/kes-policy.hcl $(kubectl get pods -l app=vault -n tenant-certmanager | grep -v NAME | awk '{print $1}'):/kes-policy.hcl -n tenant-certmanager

kubectl exec $(kubectl get pods -l app=vault -n tenant-certmanager | grep -v NAME | awk '{print $1}') -n tenant-certmanager -- sh -c 'VAULT_TOKEN='$VAULT_ROOT_TOKEN' VAULT_ADDR="http://127.0.0.1:8200" vault policy write kes-policy /kes-policy.hcl'

kubectl exec $(kubectl get pods -l app=vault -n tenant-certmanager | grep -v NAME | awk '{print $1}') -n tenant-certmanager -- sh -c 'VAULT_TOKEN='$VAULT_ROOT_TOKEN' VAULT_ADDR="http://127.0.0.1:8200" vault write auth/approle/role/kes-role token_num_uses=0 secret_id_num_uses=0 period=5m policies=kes-policy'

ROLE_ID=$(kubectl exec $(kubectl get pods -l app=vault -n tenant-certmanager | grep -v NAME | awk '{print $1}') -n tenant-certmanager -- sh -c 'VAULT_TOKEN='$VAULT_ROOT_TOKEN' VAULT_ADDR="http://127.0.0.1:8200" vault read auth/approle/role/kes-role/role-id' | grep "role_id    " | sed -e "s/role_id    //g")

SECRET_ID=$(kubectl exec $(kubectl get pods -l app=vault -n tenant-certmanager | grep -v NAME | awk '{print $1}') -n tenant-certmanager -- sh -c 'VAULT_TOKEN='$VAULT_ROOT_TOKEN' VAULT_ADDR="http://127.0.0.1:8200" vault write -f auth/approle/role/kes-role/secret-id' | grep "secret_id             " | sed -e "s/secret_id             //g")

echo " "
echo " "
echo " "
echo " "
echo "ROLE_ID = ${ROLE_ID}"
echo "SECRET_ID = ${SECRET_ID}"

• Apply the configuration:

echo "apiVersion: v1
kind: Secret
metadata:
  name: storage-certmanager-secret-kes-configuration
  namespace: tenant-certmanager
  labels:
    v1.min.io/tenant: storage-certmanager
  managedFields:
    - manager: console
      operation: Update
      apiVersion: v1
      time: '2022-11-29T14:29:22Z'
      fieldsType: FieldsV1
      fieldsV1:
        f:data:
          .: {}
          f:server-config.yaml: {}
        f:immutable: {}
        f:metadata:
          f:labels:
            .: {}
            f:v1.min.io/tenant: {}
        f:type: {}
  selfLink: /api/v1/namespaces/tenant-certmanager/secrets/storage-certmanager-secret-kes-configuration
immutable: true
type: Opaque
stringData:
  server-config.yaml: |-
    address: 0.0.0.0:7373
    root: disabled
    tls:
      key: /tmp/kes/server.key
      cert: /tmp/kes/server.crt
    policy:
      default-policy:
        paths:
        - /v1/key/create/my-minio-key
        - /v1/key/generate/my-minio-key
        - /v1/key/decrypt/my-minio-key
        identities:
        - \${MINIO_KES_IDENTITY}
    cache:
      expiry:
        any: 5m0s
        unused: 20s
    log:
      error: "on"
      audit: "off"
    keys:
      vault:
        endpoint: http://vault.tenant-certmanager.svc.cluster.local:8200
        prefix: my-minio
        approle:
          id: ${ROLE_ID}
          secret: ${SECRET_ID}
        status: {}" > kes-configuration.yaml
k apply -f kes-configuration.yaml

• Add the issuer of cert-manager:

echo "apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: tenant-certmanager-issuer
  namespace: tenant-certmanager
spec:
  selfSigned: {}" > issuer.yaml

k apply -f issuer.yaml

echo "apiVersion: cert-manager.io/v1" > certificate222.yaml
echo "kind: Certificate" >> certificate222.yaml
echo "metadata:" >> certificate222.yaml
echo "  name: tenant-certmanager-2-cert" >> certificate222.yaml
echo "  namespace: tenant-certmanager" >> certificate222.yaml
echo "spec:" >> certificate222.yaml
echo "  dnsNames:" >> certificate222.yaml
echo "    - \"*.tenant-certmanager.svc.cluster.local\"" >> certificate222.yaml
echo "    - \"*.storage-certmanager.tenant-certmanager.svc.cluster.local\"" >> certificate222.yaml
echo "    - \"*.storage-certmanager-hl.tenant-certmanager.svc.cluster.local\"" >> certificate222.yaml
echo "  secretName: tenant-certmanager-2-tls" >> certificate222.yaml
echo "  issuerRef:" >> certificate222.yaml
echo "    name: tenant-certmanager-issuer" >> certificate222.yaml

k apply -f certificate222.yaml

• Add kes in Tenant Spec with cert-manager:

k edit tenant storage-certmanager -n tenant-certmanager

spec:
  kes:
    externalCertSecret:
      name: tenant-certmanager-2-tls
      type: cert-manager.io/v1
    image: minio/kes:v0.17.6
    imagePullPolicy: IfNotPresent
    kesSecret:
      name: storage-certmanager-secret-kes-configuration
    keyName: my-minio-key
    replicas: 1
    resources: {}
    securityContext:
      fsGroup: 1000
      fsGroupChangePolicy: Always
      runAsGroup: 1000
      runAsNonRoot: true
      runAsUser: 1000

• Make sure you can encrypt:

k apply -f ~/ubuntu.yaml -n tenant-certmanager
apt update
apt install vim
apt install wget
wget https://dl.min.io/client/mc/release/linux-amd64/mc
chmod +x mc
mv mc /usr/local/bin/mc

mc alias set myminio https://minio.tenant-certmanager.svc.cluster.local minio minio123
Fingerprint of myminio public key: 4ab5e381022b304f7b80479a8ca5d2b8545b61c5f059a18e6f584008fd837700
Confirm public key y/N: y
Added `myminio` successfully.

mc mb myminio/testing
mc encrypt set sse-s3 myminio/testing/

mc encrypt info myminio/testing
Auto encryption 'sse-s3' is enabled

Additional Info:

• If Operator pod gets restarted we will face:

I0126 19:13:05.637166       1 main.go:77] Starting MinIO Operator
I0126 19:13:05.910947       1 main.go:176] caBundle on CRD updated
I0126 19:13:05.911570       1 main-controller.go:243] Setting up event handlers
I0126 19:13:05.911706       1 leaderelection.go:248] attempting to acquire leader lease minio-operator/minio-operator-lock...
I0126 19:13:05.916118       1 leaderelection.go:258] successfully acquired lease minio-operator/minio-operator-lock
I0126 19:13:05.916204       1 main-controller.go:500] minio-operator-694b5677f6-q9zcv: I am the leader, applying leader labels on myself
I0126 19:13:05.916288       1 main-controller.go:409] Waiting for API to start
I0126 19:13:05.916301       1 main-controller.go:390] Starting console TLS certificate setup
I0126 19:13:05.916304       1 main-controller.go:404] Console TLS is not enabled
I0126 19:13:05.916483       1 main-controller.go:381] Starting HTTP Upgrade Tenant Image server
I0126 19:13:05.921638       1 main-controller.go:352] Using Kubernetes CSR Version: v1
I0126 19:13:05.927228       1 main-controller.go:356] Starting HTTPS API server
I0126 19:13:05.927368       1 main-controller.go:412] Waiting for Upgrade Server to start
I0126 19:13:05.927374       1 main-controller.go:415] Waiting for Console TLS
I0126 19:13:05.927376       1 main-controller.go:419] Starting Tenant controller
I0126 19:13:05.927378       1 main-controller.go:422] Waiting for informer caches to sync
I0126 19:13:05.927396       1 main-controller.go:427] Starting workers
I0126 19:13:05.943234       1 monitoring.go:129] 'tenant-certmanager/storage-certmanager' Failed to get cluster health: Get "https://minio.tenant-certmanager.svc.cluster.local/minio/health/cluster": x509: certificate signed by unknown authority
I0126 19:13:06.045801       1 monitoring.go:129] 'tenant-certmanager/storage-certmanager' Failed to get cluster health: Get "https://minio.tenant-certmanager.svc.cluster.local/minio/health/cluster": x509: certificate signed by unknown authority
I0126 19:13:06.245798       1 monitoring.go:129] 'tenant-certmanager/storage-certmanager' Failed to get cluster health: Get "https://minio.tenant-certmanager.svc.cluster.local/minio/health/cluster": x509: certificate signed by unknown authority

• But after some time with cert-manager this will get fixed automatically, so you can ignore above issue, gets corrected after few minutes after the reboot.

• Also this is how it looks the API for custom cert in KES when using cert-manager:

• This is how it looks Tenant TLS in UI:

Conclusion:

We can safely use cert-manager to manage the certificates for us; based on my experimentation, I do recommend having different certificates, one for the tenant and other for the KES but both will be managed by cert-manager.