externalClientCertSecret for KES certificate - cniackz/public GitHub Wiki

Tested Date:

Wed Jan 25, 2023 @ 7:49 am Toronto Time

Objective:

To use self-signed certificate for KES TLS

Steps:

Create cluster & install operator

createcluster
installoperator

modify service to expose Operator via NodePort:

nodePort: 30080
type: NodePort

Deploy vault:

kubectl apply -f ~/operator/examples/vault/deployment.yaml

Wait for vault to be ready:

kubectl wait --namespace default \
	--for=condition=ready pod \
	--selector=app=vault \
	--timeout=120s

Get Vault Token:

VAULT_ROOT_TOKEN=$(kubectl logs -l app=vault | grep "Root Token: " | sed -e "s/Root Token: //g")

Enabled approle auth method:

kubectl exec $(kubectl get pods -l app=vault  | grep -v NAME | awk '{print $1}') -- sh -c 'VAULT_TOKEN='$VAULT_ROOT_TOKEN' VAULT_ADDR="http://127.0.0.1:8200" vault auth enable approle'

Enabled the kv secrets engine:

kubectl exec $(kubectl get pods -l app=vault  | grep -v NAME | awk '{print $1}') -- sh -c 'VAULT_TOKEN='$VAULT_ROOT_TOKEN' VAULT_ADDR="http://127.0.0.1:8200" vault secrets enable kv'

copy kes file:

kubectl cp ~/operator/examples/vault/kes-policy.hcl $(kubectl get pods -l app=vault  | grep -v NAME | awk '{print $1}'):/kes-policy.hcl

Upload policy:

kubectl exec $(kubectl get pods -l app=vault  | grep -v NAME | awk '{print $1}') -- sh -c 'VAULT_TOKEN='$VAULT_ROOT_TOKEN' VAULT_ADDR="http://127.0.0.1:8200" vault policy write kes-policy /kes-policy.hcl'

Write data to auth/approle/role/kes-role:

kubectl exec $(kubectl get pods -l app=vault  | grep -v NAME | awk '{print $1}') -- sh -c 'VAULT_TOKEN='$VAULT_ROOT_TOKEN' VAULT_ADDR="http://127.0.0.1:8200" vault write auth/approle/role/kes-role token_num_uses=0 secret_id_num_uses=0 period=5m policies=kes-policy'

Get Role ID

ROLE_ID=$(kubectl exec $(kubectl get pods -l app=vault  | grep -v NAME | awk '{print $1}') -- sh -c 'VAULT_TOKEN='$VAULT_ROOT_TOKEN' VAULT_ADDR="http://127.0.0.1:8200" vault read auth/approle/role/kes-role/role-id' | grep "role_id    " | sed -e "s/role_id    //g")

Get Secret ID

SECRET_ID=$(kubectl exec $(kubectl get pods -l app=vault  | grep -v NAME | awk '{print $1}') -- sh -c 'VAULT_TOKEN='$VAULT_ROOT_TOKEN' VAULT_ADDR="http://127.0.0.1:8200" vault write -f auth/approle/role/kes-role/secret-id' | grep "secret_id             " | sed -e "s/secret_id             //g")

Get token

SA_TOKEN=$(kubectl -n minio-operator  get secret console-sa-secret -o jsonpath="{.data.token}" | base64 --decode);
echo "SA_TOKEN: ${SA_TOKEN}";

Get cookie, notice I am using port 30080 because I am using NodePort in the service:

COOKIE=$(curl 'http://localhost:30080/api/v1/login/operator' -X POST -H 'Content-Type: application/json' --data-raw '{"jwt":"'$SA_TOKEN'"}' -i | grep "Set-Cookie: token=" | sed -e "s/Set-Cookie: token=//g" | awk -F ';' '{print $1}');
echo $COOKIE;

Create the tenant:

CREDENTIALS=$(curl 'http://localhost:30080/api/v1/tenants' -X POST -H 'Content-Type: application/json' -H 'Cookie: token='$COOKIE'' --data-raw '{"name":"kes-tenant","namespace":"default","access_key":"","secret_key":"","access_keys":[],"secret_keys":[],"enable_tls":true,"enable_console":true,"enable_prometheus":true,"service_name":"","image":"","expose_minio":true,"expose_console":true,"pools":[{"name":"pool-0","servers":4,"volumes_per_server":1,"volume_configuration":{"size":26843545600,"storage_class_name":"standard"},"securityContext":null,"affinity":{"podAntiAffinity":{"requiredDuringSchedulingIgnoredDuringExecution":[{"labelSelector":{"matchExpressions":[{"key":"v1.min.io/tenant","operator":"In","values":["kes-tenant"]},{"key":"v1.min.io/pool","operator":"In","values":["pool-0"]}]},"topologyKey":"kubernetes.io/hostname"}]}}}],"erasureCodingParity":2,"logSearchConfiguration":{"image":"minio/operator:dev","postgres_image":"","postgres_init_image":""},"prometheusConfiguration":{"image":"","sidecar_image":"","init_image":""},"tls":{"minio":[],"ca_certificates":[],"console_ca_certificates":[]},"encryption":{"replicas":"1","securityContext":{"runAsUser":"1000","runAsGroup":"1000","fsGroup":"1000","runAsNonRoot":true},"image":"","vault":{"endpoint":"http://vault.default.svc.cluster.local:8200","engine":"","namespace":"","prefix":"my-minio","approle":{"engine":"","id":"'$ROLE_ID'","secret":"'$SECRET_ID'","retry":0},"tls":{},"status":{"ping":0}}},"idp":{"keys":[{"access_key":"console","secret_key":"console123"}]}}')

Disable Audit Logs and Prometheus:

Follow steps from https://github.com/cniackz/public/wiki/Create-Cert-for-KES-manually-self-signed-cert and created new cert that expires in 10 minutes and waited 10 minutes.

10 minutes is 600 seconds

k apply -f ~/minio/ubuntu.yaml

apt update
apt install openssl
apt install -y vim
openssl genrsa -out private.key 2048
touch cert.cnf
vi cert.cnf

[req]
distinguished_name = req_distinguished_name
req_extensions = req_ext
prompt = no

[req_distinguished_name]
O = "system:nodes"
C = US
CN  = "system:node:*.kes-tenant-kes-hl-svc.default.svc.cluster.local"

[req_ext]
subjectAltName = @alt_names

[alt_names]
DNS.1 = kes-tenant-kes-0.kes-tenant-kes-hl-svc.default.svc.cluster.local
DNS.2 = kes-tenant-kes-hl-svc.default.svc.cluster.local

openssl req -new -config cert.cnf -key private.key -out kes.csr
cat kes.csr | base64 | tr -d "\n"

In your laptop with name: csr.yaml

apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
  name: kes-csr 
spec:
  expirationSeconds: 600
  groups:
  - system:serviceaccounts
  - system:serviceaccounts:minio-operator
  - system:authenticated
  - system:nodes
  request: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJRFF6Q0NBaXNDQVFBd2JERVZNQk1HQTFVRUNnd01jM2x6ZEdWdE9tNXZaR1Z6TVFzd0NRWURWUVFHRXdKVgpVekZHTUVRR0ExVUVBd3c5YzNsemRHVnRPbTV2WkdVNktpNXJaWE10ZEdWdVlXNTBMV3RsY3kxb2JDMXpkbU11ClpHVm1ZWFZzZEM1emRtTXVZMngxYzNSbGNpNXNiMk5oYkRDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVAKQURDQ0FRb0NnZ0VCQU1SWnI2aWREcWM5eGVZWkprZEVLdXVvc1dXdjEwNzdJMHdpSnhZNzFiMitrRmpiZ2FHVQplZmtLNGlqdlBUTFlNU3JxMkpNa1RidWFYSnRqbWZhUGMvd3BJODMybkxqcWpBVExia3hwdEFLRHoyQUNiWXF4CnoxVXJteUQyRm1CUmJ5ZlRiRVloR3d2MEE1NjZBMnZHK0hLNDlUcjJNUUxPQXk5bGYwdUlSd1RWVXF3NlUrazUKQnpvMlNLTDdlTnZOb0ZCMm1TQWVmKzhUQkRLM2VNYUE3cmRmVDc4blcxVlgyNWVEUmMwRzd2SzJ1TjIxbmhwWAo5ZDV4ZGZsSTB1ZEZrUktMWkdGWmk3Z3ZXdHlGSlBwY2llc01GdzN6bzZKUSt6OHdQTHZmK3Q3RGZtUGpKTlBWCjNzSGpNT0pCRDFJekYyZFBJWDI3QXlIMjg2T2V2SkF1Yzc4Q0F3RUFBYUNCa1RDQmpnWUpLb1pJaHZjTkFRa08KTVlHQU1INHdmQVlEVlIwUkJIVXdjNEpBYTJWekxYUmxibUZ1ZEMxclpYTXRNQzVyWlhNdGRHVnVZVzUwTFd0bApjeTFvYkMxemRtTXVaR1ZtWVhWc2RDNXpkbU11WTJ4MWMzUmxjaTVzYjJOaGJJSXZhMlZ6TFhSbGJtRnVkQzFyClpYTXRhR3d0YzNaakxtUmxabUYxYkhRdWMzWmpMbU5zZFhOMFpYSXViRzlqWVd3d0RRWUpLb1pJaHZjTkFRRUwKQlFBRGdnRUJBSWh0OGJ0RFg3NVQ5NjNDcllBRVA4bFJrUEF4SXF3ZGtjc0c0UGxZZ2xjQmtpbVZSYlFwWDZlMgp0a0FrWlc1SFVzV3lmbVprRXA4RXRHaWhTT2FTN1FQQ3Y2azBWY2xJR1k4KzBIbEg5RHM5bGRUcXNvbm9xNDk1ClJMSUtNZmg2bXFDakZJeHB4ZzF3cFJ5cjNmSXZzNjNjNDVaMm81ME1aU1h6ejNVY0c4RWNVejd2cm5QTnBtSnEKTHhOS0tkR1BBcG9WWGRCOWhvWkxsamthNDdDd1pDTzVHeVNJREZYMWZ2L1ExRTFjbkVDNmpnT0xieGkxZ0RQOApORW1WOU9jVU8xS0U0VGIxM2dxNmpVVXVMT1RlN2tWaDdKNk1GY1Fnd0krMndtWmNyMmdJd3ZXaDlPeTg4S2ZFCi94QzVobmdrMVdmaTNOdldQYXVpNkFZNjlaLzhhSnc9Ci0tLS0tRU5EIENFUlRJRklDQVRFIFJFUVVFU1QtLS0tLQo=
  signerName: kubernetes.io/kubelet-serving
  usages:
  - digital signature
  - key encipherment
  - server auth
  username: system:serviceaccount:minio-operator:minio-operator

k apply -f csr.yaml
kubectl certificate approve kes-csr
k get csr kes-csr -o jsonpath='{.status.certificate}'| base64 -d > public.crt

Modify the secret in the cluster and add two things [copy secret, delete secret, create secret with new values]: kes-tenant-kes-tls

> The private.key from Ubuntu machine inside the cluster
> The public.crt from your laptop

touch secret.yaml
subl secret.yaml
k delete secret kes-tenant-kes-tls -n default
k apply -f secret.yaml

Wait 10 minutes
Look at kes-tenant-kes-0 pod and it should display TLS issue

Authenticating to Hashicorp Vault 'http://vault.default.svc.cluster.local:8200' ... 
Endpoint: https://127.0.0.1:7373        https://10.244.4.3:7373       

Admin:    _     [ disabled ]
Auth:     off   [ any client can connect but policies still apply ]

Keys:     Hashicorp Vault: http://vault.default.svc.cluster.local:8200

CLI:      export KES_SERVER=https://127.0.0.1:7373
          export KES_CLIENT_KEY=<client-private-key>   // e.g. $HOME/root.key
          export KES_CLIENT_CERT=<client-certificate>  // e.g. $HOME/root.cert
          kes --help
{"message":"2023/01/25 16:23:48 http: TLS handshake error from 10.244.1.4:39228: remote error: tls: bad certificate"}
{"message":"2023/01/25 16:23:54 http: TLS handshake error from 10.244.3.4:45336: remote error: tls: bad certificate"}

Look at MinIO Pods and they should also expose TLS issues after cert expired.

API: AssumeRole()
Time: 16:23:54 UTC 01/25/2023
DeploymentID: 5ddb29ba-6663-4087-8940-06814188f89c
RequestID: 173D99E27A0D80CF
RemoteHost: 10.244.1.4
Host: kes-tenant-pool-0-3.kes-tenant-hl.default.svc.cluster.local:9000
UserAgent: Go-http-client/1.1
Error: Post "https://kes-tenant-kes-hl-svc.default.svc.cluster.local:7373/v1/key/generate/my-minio-key": x509: certificate has expired or is not yet valid: current time 2023-01-25T16:23:54Z is after 2023-01-25T16:22:00Z (*url.Error)
       4: internal/logger/logger.go:258:logger.LogIf()
       3: cmd/sts-errors.go:53:cmd.writeSTSErrorResponse()
       2: cmd/sts-handlers.go:279:cmd.(*stsAPIHandlers).AssumeRole()
       1: net/http/server.go:2109:http.HandlerFunc.ServeHTTP()

Create new certificate and increase the expiration to a week so expirationSeconds: 604800 and above issue should be fixed, when repeating above steps, you need a new name for CSR:

apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
  name: kes-csr-2 <----- Example

A you need to approve new kes-csr-2 to obtain new public.crt that goes in the new secret so that it can work the regeneration.

Cesars-MacBook-Pro:~ cniackz$ k apply -f csr.yaml
certificatesigningrequest.certificates.k8s.io/kes-csr-2 created
Cesars-MacBook-Pro:~ cniackz$ kubectl certificate approve kes-csr-2
certificatesigningrequest.certificates.k8s.io/kes-csr-2 approved
Cesars-MacBook-Pro:~ cniackz$ k get csr kes-csr-2 -o jsonpath='{.status.certificate}'| base64 -d > public.crt

See issues are gone in kes pod:

Authenticating to Hashicorp Vault 'http://vault.default.svc.cluster.local:8200' ... 
Endpoint: https://127.0.0.1:7373        https://10.244.4.11:7373      

Admin:    _     [ disabled ]
Auth:     off   [ any client can connect but policies still apply ]

Keys:     Hashicorp Vault: http://vault.default.svc.cluster.local:8200

CLI:      export KES_SERVER=https://127.0.0.1:7373
          export KES_CLIENT_KEY=<client-private-key>   // e.g. $HOME/root.key
          export KES_CLIENT_CERT=<client-certificate>  // e.g. $HOME/root.cert
          kes --help

Same for MinIO if you restart the pod:

Waiting for all MinIO sub-systems to be initialized.. lock acquired
Automatically configured API requests per node based on available memory on the system: 224
All MinIO sub-systems initialized successfully in 186.749625ms
MinIO Object Storage Server
Copyright: 2015-2023 MinIO, Inc.
License: GNU AGPLv3 <https://www.gnu.org/licenses/agpl-3.0.html>
Version: RELEASE.2023-01-25T00-19-54Z (go1.19.4 linux/arm64)

Status:         4 Online, 0 Offline. 
API: https://minio.default.svc.cluster.local 
Console: https://10.244.2.7:9443 https://127.0.0.1:9443   

Documentation: https://min.io/docs/minio/linux/index.html

Now that you gain enough confidence in this process, provide custom certificates for KES as Kirill did and learn how that is done

kes-tenant-client-tls

private.key

-----BEGIN PRIVATE KEY-----
MC4CAQAwBQYDK2VwBCIEIEzw6/W8lYZzU5rWMfKeeqW4WaYTJxcqUoJqjNt5ojNG
-----END PRIVATE KEY-----

public.crt

-----BEGIN CERTIFICATE-----
MIIBojCCAVSgAwIBAgIRAKNwOjoceP+xUIxkVePkcZ8wBQYDK2VwMCoxKDAmBgNV
BAMTH21pbmlvLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwwHhcNMjMwMTI1MTYw
MjA3WhcNMjQwMTI1MTYwMjA3WjAqMSgwJgYDVQQDEx9taW5pby5kZWZhdWx0LnN2
Yy5jbHVzdGVyLmxvY2FsMCowBQYDK2VwAyEAUJ9uXEotu1bU5aVir66BsAcmzzIf
I8rlYnDNeggj8z+jgY4wgYswDgYDVR0PAQH/BAQDAgeAMB0GA1UdJQQWMBQGCCsG
AQUFBwMBBggrBgEFBQcDAjAMBgNVHRMBAf8EAjAAMEwGA1UdEQRFMEOCQWtlcy10
ZW5hbnQtcG9vbC0wLXswLi4uM30ua2VzLXRlbmFudC1obC5kZWZhdWx0LnN2Yy5j
bHVzdGVyLmxvY2FsMAUGAytlcANBAKfyivFIPCV4UFEc0nYhlSx4dquftxa5AiFr
siaBEOaLEp9Bq9Bpl02DiEWOkq93ajhQxX4fk7mmDXAvLbZ3QwI=
-----END CERTIFICATE-----

kes-tenant-kes-tls

kes-tenant-tls