MinIO TLS Process certificate k8s and OpenShift - cniackz/public GitHub Wiki

Relevant information:

Steps:

  1. Create the private key:
openssl genrsa -out private.key 2048
-----BEGIN PRIVATE KEY-----
MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQDwtLnkWgEjpvnm
lRJnRBbTjFy966DnITGzQnZl01z61NWZChzTWp8OxlOrmr5MpFzK7UFFvnJrbSRp
ZZ63EUAIIBwGnI7BooOLDzDg699JDoWZKgCGV17lKMCXMgvS7vR93nQ4QKwJCX0A
7UZc/J0kVs3PpkYI3eY+dVy8aFv9FxYze3sE3qrPeDIxLxRDQCs+OYECp8EnaD2h
Yz5FSZjWA6NOCZX2Ak7YQ8sqW0Oyvl8FNqajliCnAZE+59UDGckVxUMKT9f6hQCp
FmPW/5wOaiV+dWLgdJfw8//sn6j7OOJ//r3lUnAWgP2cdd9Smc18nP+1+6RtK1xA
j5XLSWtDAgMBAAECggEAO46D9je802zzpTVaRdnHihVPNyCvgbCpYvC9acln1ppF
tWWWp0bjVx0EQ8/4aO88FGuqvE54chMhsMBXinVGshv7wzkGa2vB/2zFS8mgyCtD
LBgD5DmG/c1dBav41/qO3PvnJqn9xwileM68m7JXUU/3AXtjFbuNxgN07BsoM5EB
aMYMIV3c2ro8vZKI4iAtfJHwefYFBk0LvLoPzXlG7RJQ5YbPeNqfsLdP5gpy8W2/
bi9VxvfWQVOcAahrpN/Kid6xFCpLee3dwuD7UXCfYa/ZWk5+aSPGNEHSnAR+5N6n
jco5gJGL0WtlVBAIjgzqSeiDt0mgNncK7KngglpuUQKBgQD+dgQ8lKRatYgVjtG0
uq94eFwiN2og+RWOq0PTnzv3ZodEijtWVZofpKMicVQrdlsRaq96ahgyXgCTTKmo
/H6Le1wdA73vniT/U4DPVE80k+Wf/LnBXI46X151E6AZ2KGoN/hozKvA3d5L8TvA
9CIyTthMpFP9z6wgKzLeKUFTswKBgQDyKWme9K8Sn7f27+BYiYDsy6LckMW0gpAw
vrpggzmdpBnttsspgYJ0L+TRt4vNsqHa2q8FScRSuMfSF7cfQc9fNDs1R0zV8Pvl
AAFLUEnycSTrpgfPNaupo7074bwj/Qgl4owL+PUL/QLb3HpW+y5S4iJbJvpsijt+
LIqJk9sCMQKBgEbx4HyDdM9VjXt1NkKNSh7sb8Oab5yB1eDRHzgBTVbM3ync4G5R
r4bjABi6QB1eAkX42LPooJZqXv6ygRr8SsvYuPXd9Ta5gNcvENq9RtgB5COW8rNW
etELhepebmTGYUvlTvzpE3nSYggKWf6SEna0RNoZdtFs9TCC2svV9MDBAoGAP3/O
e2kL82VAVxdFi9Hapb0sIowv24qVBsMsmhwu4oBicCwLkAGu1IquhLONjE24FWxc
kpVqToq3kwa29RJLr1pB4FGm+z6qvrzGmhoemd1DYityRsp7DguqlmY18HXsSgri
0Ezi7NbWhhKFXgOVfRRhD+fjqMyCly5U0Yic2DECgYEAsfFRr0+r6+nHIppIjruA
MwG5LXw50oIjcLqPaMx66dWOC1W7C0FP6bnOFMxdJLCYxC4wZelFcvN2OcfJ4utN
7BEzgn35KNg8FtO6SEnPQbPevCunFsm+6Htu0mdvUHgXkjn/KfK6upIsFlIVhZsE
uNIpulCOnZTTLtdtOG1Z3/E=
-----END PRIVATE KEY-----
  1. cert.cnf with data from my tenant:
  • File: cert.cnf
[req]
distinguished_name = req_distinguished_name
req_extensions = req_ext
prompt = no

[req_distinguished_name]
O = "system:nodes"
C = US
CN  = "system:node:*.pepe-hl.pepe.svc.cluster.local"

[req_ext]
subjectAltName = @alt_names

[alt_names]
DNS.1 = pepe-pool-0-0.pepe-hl.pepe.svc.cluster.local
DNS.2 = minio.pepe.svc.cluster.local
DNS.3 = minio.pepe
DNS.4 = .pepe.svc
DNS.5 = *.
DNS.6 = *.pepe.svc.cluster.local
  1. Get the tenant.csr
openssl req -new -config cert.cnf -key private.key -out tenant.csr
                             |            |_ first openssl command (private key)
                             |__ file above
-----BEGIN CERTIFICATE REQUEST-----
MIIDRDCCAiwCAQAwWzEVMBMGA1UECgwMc3lzdGVtOm5vZGVzMQswCQYDVQQGEwJV
UzE1MDMGA1UEAwwsc3lzdGVtOm5vZGU6Ki5wZXBlLWhsLnBlcGUuc3ZjLmNsdXN0
ZXIubG9jYWwwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDwtLnkWgEj
pvnmlRJnRBbTjFy966DnITGzQnZl01z61NWZChzTWp8OxlOrmr5MpFzK7UFFvnJr
bSRpZZ63EUAIIBwGnI7BooOLDzDg699JDoWZKgCGV17lKMCXMgvS7vR93nQ4QKwJ
CX0A7UZc/J0kVs3PpkYI3eY+dVy8aFv9FxYze3sE3qrPeDIxLxRDQCs+OYECp8En
aD2hYz5FSZjWA6NOCZX2Ak7YQ8sqW0Oyvl8FNqajliCnAZE+59UDGckVxUMKT9f6
hQCpFmPW/5wOaiV+dWLgdJfw8//sn6j7OOJ//r3lUnAWgP2cdd9Smc18nP+1+6Rt
K1xAj5XLSWtDAgMBAAGggaMwgaAGCSqGSIb3DQEJDjGBkjCBjzCBjAYDVR0RBIGE
MIGBgixwZXBlLXBvb2wtMC0wLnBlcGUtaGwucGVwZS5zdmMuY2x1c3Rlci5sb2Nh
bIIcbWluaW8ucGVwZS5zdmMuY2x1c3Rlci5sb2NhbIIKbWluaW8ucGVwZYIJLnBl
cGUuc3ZjggIqLoIYKi5wZXBlLnN2Yy5jbHVzdGVyLmxvY2FsMA0GCSqGSIb3DQEB
CwUAA4IBAQDs5socUlqZJPftjGDFpl2GUU4e2S2i4MYOqS84w63cx//3Ln5ljp8F
B+D4pft5VpogABUSHaoNJqJbMeNqLvMuCF6tDjwg4PYNXCiURfQbcLV+b8crLRmR
Y3iJufORGYZA5VRFuJZZfHfqCZhP2W1RxKlE2mngmakJ9VP5z3fI8nYzQOd2ckGz
f3QVPJds2iarqK2TAdQQoAfoNVNcUz3Jeh/VDrdnsj8/bmPd8zLsdvOVGxQcQavK
83cDZz27W9ScC6Gmc4v4qTaO1Ov6McjSKPYlddSIvQwOG+UNBjdJSgge3hpQBocg
ioU2XA+i0OWWk0Lpw+xvPz9GERGedKcR
-----END CERTIFICATE REQUEST-----
  1. Encode:
cat tenant.csr | base64 | tr -d "\n"

Expected:

LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJRFJEQ0NBaXdDQVFBd1d6RVZNQk1HQTFVRUNnd01jM2x6ZEdWdE9tNXZaR1Z6TVFzd0NRWURWUVFHRXdKVgpVekUxTURNR0ExVUVBd3dzYzNsemRHVnRPbTV2WkdVNktpNXdaWEJsTFdoc0xuQmxjR1V1YzNaakxtTnNkWE4wClpYSXViRzlqWVd3d2dnRWlNQTBHQ1NxR1NJYjNEUUVCQVFVQUE0SUJEd0F3Z2dFS0FvSUJBUUR3dExua1dnRWoKcHZubWxSSm5SQmJUakZ5OTY2RG5JVEd6UW5abDAxejYxTldaQ2h6VFdwOE94bE9ybXI1TXBGeks3VUZGdm5KcgpiU1JwWlo2M0VVQUlJQndHbkk3Qm9vT0xEekRnNjk5SkRvV1pLZ0NHVjE3bEtNQ1hNZ3ZTN3ZSOTNuUTRRS3dKCkNYMEE3VVpjL0owa1ZzM1Bwa1lJM2VZK2RWeThhRnY5RnhZemUzc0UzcXJQZURJeEx4UkRRQ3MrT1lFQ3A4RW4KYUQyaFl6NUZTWmpXQTZOT0NaWDJBazdZUThzcVcwT3l2bDhGTnFhamxpQ25BWkUrNTlVREdja1Z4VU1LVDlmNgpoUUNwRm1QVy81d09haVYrZFdMZ2RKZnc4Ly9zbjZqN09PSi8vcjNsVW5BV2dQMmNkZDlTbWMxOG5QKzErNlJ0CksxeEFqNVhMU1d0REFnTUJBQUdnZ2FNd2dhQUdDU3FHU0liM0RRRUpEakdCa2pDQmp6Q0JqQVlEVlIwUkJJR0UKTUlHQmdpeHdaWEJsTFhCdmIyd3RNQzB3TG5CbGNHVXRhR3d1Y0dWd1pTNXpkbU11WTJ4MWMzUmxjaTVzYjJOaApiSUljYldsdWFXOHVjR1Z3WlM1emRtTXVZMngxYzNSbGNpNXNiMk5oYklJS2JXbHVhVzh1Y0dWd1pZSUpMbkJsCmNHVXVjM1pqZ2dJcUxvSVlLaTV3WlhCbExuTjJZeTVqYkhWemRHVnlMbXh2WTJGc01BMEdDU3FHU0liM0RRRUIKQ3dVQUE0SUJBUURzNXNvY1VscVpKUGZ0akdERnBsMkdVVTRlMlMyaTRNWU9xUzg0dzYzY3gvLzNMbjVsanA4RgpCK0Q0cGZ0NVZwb2dBQlVTSGFvTkpxSmJNZU5xTHZNdUNGNnREandnNFBZTlhDaVVSZlFiY0xWK2I4Y3JMUm1SClkzaUp1Zk9SR1laQTVWUkZ1SlpaZkhmcUNaaFAyVzFSeEtsRTJtbmdtYWtKOVZQNXozZkk4bll6UU9kMmNrR3oKZjNRVlBKZHMyaWFycUsyVEFkUVFvQWZvTlZOY1V6M0plaC9WRHJkbnNqOC9ibVBkOHpMc2R2T1ZHeFFjUWF2Swo4M2NEWnoyN1c5U2NDNkdtYzR2NHFUYU8xT3Y2TWNqU0tQWWxkZFNJdlF3T0crVU5CamRKU2dnZTNocFFCb2NnCmlvVTJYQStpME9XV2swTHB3K3h2UHo5R0VSR2VkS2NSCi0tLS0tRU5EIENFUlRJRklDQVRFIFJFUVVFU1QtLS0tLQo=
  1. Create CSR with above encoded message:
subl /Users/cniackz/minio/tenant.yaml
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
  name: tenant-csr 
spec:
  groups:
  - system:serviceaccounts
  - system:serviceaccounts:minio-operator
  - system:authenticated
  - system:nodes
  request: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJRFJEQ0NBaXdDQVFBd1d6RVZNQk1HQTFVRUNnd01jM2x6ZEdWdE9tNXZaR1Z6TVFzd0NRWURWUVFHRXdKVgpVekUxTURNR0ExVUVBd3dzYzNsemRHVnRPbTV2WkdVNktpNXdaWEJsTFdoc0xuQmxjR1V1YzNaakxtTnNkWE4wClpYSXViRzlqWVd3d2dnRWlNQTBHQ1NxR1NJYjNEUUVCQVFVQUE0SUJEd0F3Z2dFS0FvSUJBUURuT2RmWHYvUDUKMFBUTnFPOXZkWDRvSWRDWFJNOHpvS0pSSVN0WVJsKzU0L3NzNlIxYzFnZCt5SkF1MmJaNzlobkE2aTdJOE84ZQprVUgrcmY4Tk11czdPMkF5QitPQjZaaDFqQlFJK2JTdk4wZndQaDFwekhkUE4rZHlReXdTV3E0Yld2WVJiQmJCCnRIWnBwUFFBWjMyaXZ0WVVsc3BOOHBuV2VETDI4SjNUZ2dhRFlzcnE0VW1Qb2lBUVZSN2pNUmJGV0E2SXZHM1kKYmh5VVI4dHVLU0NRNnYzdnI0WEt3ZnZ6bGRzZUJvZDlza3F3RVdXUjhjWGFkRlNXd1FjUm9OenpMTkc1aWpZYQpaQUNrd3ZXSGd5THBYQTNhZEl0MW9hblM4dXR2OHFNVTdoTWQrcmwreC92eHAwZ1gyYTdaaHdNTFFKcll4R2tmCm90eXc5MU1UejVGckFnTUJBQUdnZ2FNd2dhQUdDU3FHU0liM0RRRUpEakdCa2pDQmp6Q0JqQVlEVlIwUkJJR0UKTUlHQmdpeHdaWEJsTFhCdmIyd3RNQzB3TG5CbGNHVXRhR3d1Y0dWd1pTNXpkbU11WTJ4MWMzUmxjaTVzYjJOaApiSUljYldsdWFXOHVjR1Z3WlM1emRtTXVZMngxYzNSbGNpNXNiMk5oYklJS2JXbHVhVzh1Y0dWd1pZSUpMbkJsCmNHVXVjM1pqZ2dJcUxvSVlLaTV3WlhCbExuTjJZeTVqYkhWemRHVnlMbXh2WTJGc01BMEdDU3FHU0liM0RRRUIKQ3dVQUE0SUJBUUJEZUhRdktyOVE3NUdyOUN1OG9RSUFmelhvMXZreCtnZjhONGJTdWlGUnFLWEdMUDA3Mk54WAo2MXY4bW41V0krd0lMazdtMTA2MWQvaWtFc1F3R3pJL0lTbGIvcGdDeXQrNlg5MWFsQkMyd3NaVXBwKzJaeVZqClhDQlpESy84MDgzOHJseUU1dlR5TTVNQ1Q0ZlJkU1BDT3ZQQ0dtclFxNXpuZnRtbFdrYUM2VE1nbEhWc01MUXcKUVlFWjN2MmhualMxYVpIc1d4SEFuLys0YXF0cnl0UDZVWndxQStqUnFsS3ZUVURUSDJ0YnhJRkl2dVBocGtNdApwYW9IRTdiMGVRMm9rcm5FQk9UUUJPZm5aaC90MzFDOG5sdStVNFNJUkZ1S2JKVGlWZ3JrSGpMQWljS3p4YlEwCk4wTXV1d0JJT0xkdzllRmdGckRYd1ZjSXJsRnFwY0JVCi0tLS0tRU5EIENFUlRJRklDQVRFIFJFUVVFU1QtLS0tLQo=
  signerName: kubernetes.io/kubelet-serving
  usages:
  - digital signature
  - key encipherment
  - server auth
  username: system:serviceaccount:minio-operator:minio-operator
  1. Apply it:
oc apply -f /Users/cniackz/minio/tenant.yaml

Expected:

$ oc apply -f /Users/cniackz/minio/tenant.yaml
certificatesigningrequest.certificates.k8s.io/tenant-csr created
  1. Approve it:
  • k8s:
kubectl certificate approve tenant-csr
oc adm certificate approve tenant-csr

Expected:

$ kubectl certificate approve tenant-csr
certificatesigningrequest.certificates.k8s.io/tenant-csr approved

$ oc get csr
NAME         AGE   SIGNERNAME                      REQUESTOR   REQUESTEDDURATION   CONDITION
tenant-csr   56s   kubernetes.io/kubelet-serving   kubeadmin   <none>              Approved,Issued
  1. Get the public cert:
oc get csr tenant-csr -o jsonpath='{.status.certificate}'| base64 -d > public.crt
-----BEGIN CERTIFICATE-----
MIID8zCCAtugAwIBAgIRAO+MAOD0ZT4M1CEHABE0+ZowDQYJKoZIhvcNAQELBQAw
JjEkMCIGA1UEAwwba3ViZS1jc3Itc2lnbmVyX0AxNjY1MDM0NjQ1MB4XDTIyMTEw
OTIyMjIyNVoXDTIzMTEwOTIyMjIyNVowWzELMAkGA1UEBhMCVVMxFTATBgNVBAoT
DHN5c3RlbTpub2RlczE1MDMGA1UEAwwsc3lzdGVtOm5vZGU6Ki5wZXBlLWhsLnBl
cGUuc3ZjLmNsdXN0ZXIubG9jYWwwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK
AoIBAQDwtLnkWgEjpvnmlRJnRBbTjFy966DnITGzQnZl01z61NWZChzTWp8OxlOr
mr5MpFzK7UFFvnJrbSRpZZ63EUAIIBwGnI7BooOLDzDg699JDoWZKgCGV17lKMCX
MgvS7vR93nQ4QKwJCX0A7UZc/J0kVs3PpkYI3eY+dVy8aFv9FxYze3sE3qrPeDIx
LxRDQCs+OYECp8EnaD2hYz5FSZjWA6NOCZX2Ak7YQ8sqW0Oyvl8FNqajliCnAZE+
59UDGckVxUMKT9f6hQCpFmPW/5wOaiV+dWLgdJfw8//sn6j7OOJ//r3lUnAWgP2c
dd9Smc18nP+1+6RtK1xAj5XLSWtDAgMBAAGjgeYwgeMwDgYDVR0PAQH/BAQDAgWg
MBMGA1UdJQQMMAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAU
hO/V6yEcA24VA2rU2has0C+1SGgwgYwGA1UdEQSBhDCBgYIscGVwZS1wb29sLTAt
MC5wZXBlLWhsLnBlcGUuc3ZjLmNsdXN0ZXIubG9jYWyCHG1pbmlvLnBlcGUuc3Zj
LmNsdXN0ZXIubG9jYWyCCm1pbmlvLnBlcGWCCS5wZXBlLnN2Y4ICKi6CGCoucGVw
ZS5zdmMuY2x1c3Rlci5sb2NhbDANBgkqhkiG9w0BAQsFAAOCAQEAl3xo41knf/gQ
0sgS5iBAcgQl3gylcI6uzyXk3T85cmVS44ebLDZ3cGtpuSGBoGApVFDRqcBJjPj3
kOJZLOgO9KMxCM7HrMvf74cEkIjGEaAJSaTbkXKEB1JN4ZHfrJocmpnFsRUC6Sx/
JISkrZActjFYdwZ1KKQVIzpLE0gAxlCswsdtsmEHCEeeSSqUb0efuhbIdeV4c4hC
+npnVrRjJOY05bNel5thTe5STQqE5li0TVQLnN4SeOdpDgYVa4arggWAFt/EMX5r
gsCpKP+KDLYCej4WK9V2BvtTdz3LUep9LjtKfpEj9bl8orJj8z8h6lCb9G9lggIM
+H0J4+24Sw==
-----END CERTIFICATE-----

================================

================================ PAY CLOSE ATTENTION TO THIS PART: For OpenShift, we need the Wildcard Certificate and the Signer in /var/run/secrets/kubernetes.io/serviceaccount/ca.crt. In Other words, above generated cert is not enough and has to be concatenated to the Signer then a WorkAround is needed to actually patch the proxy and put the cert in ca.crt.

# Steps obtained from: https://access.redhat.com/solutions/6013471
# How to add a custom CA/CA-chain to "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
# To add the CA/CA-chain to the pod level mounted CA file, which is /var/run/secrets/kubernetes.io/serviceaccount/ca.crt , the [custom ingress certificate configuration steps](https://docs.openshift.com/container-platform/4.7/security/certificates/replacing-default-ingress-certificate.html) can be used.

# First generate the certificate of the signer:
oc get secret csr-signer -n openshift-kube-controller-manager-operator -o template='{{ index .data "tls.crt"}}' | base64 -d > route-ca.crt

# Then, put together the above cert along with its signer in a file called ingress.pem
cat public.crt route-ca.crt > ingress.pem

# Create a secret using the ingress.pem file above and the private.key from step 1
oc create secret tls secretocuatro --cert=ingress.pem --key=private.key -n openshift-ingress

# Patch it, and wait for couple of minutes for the cert to be located at /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
oc patch ingresscontroller.operator default --type=merge -p '{"spec":{"defaultCertificate": {"name": "secretocuatro"}}}' -n openshift-ingress-operator

================================

================================

  1. Verify it:
openssl verify -verbose -CAfile /var/run/secrets/kubernetes.io/serviceaccount/ca.crt public.crt
                                                                                          |__ File generated step above

Expected in k8s is:

root@ubuntu:/tmp# openssl verify -verbose -CAfile /var/run/secrets/kubernetes.io/serviceaccount/ca.crt public.crt 
public.crt: OK

Expected in OpenShift:

root@ubuntu:/attempt1# openssl verify -verbose -CAfile /var/run/secrets/kubernetes.io/serviceaccount/ca.crt public.crt
public.crt: OK
  1. Deploy Operator

  2. Deploy Tenant

  3. Tenant can be deployed and Operator pod can trust its certificate because we helped operator:

Screenshot 2022-11-11 at 8 55 46 AM
sh-4.4$ curl https://minio.pepe.svc.cluster.local
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
sh-4.4$ 
sh-4.4$ 
sh-4.4$ 
sh-4.4$ curl --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt https://minio.pepe.svc.cluster.local
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied.</Message><Resource>/</Resource><RequestId>17268F12146D2618</RequestId><HostId>a14d363c-841a-4a5f-b879-3218f17ab6d6</HostId></Error>sh-4.4$ 
sh-4.4$ 
sh-4.4$ 
sh-4.4$ 
sh-4.4$ 
sh-4.4$ 
image
⚠️ **GitHub.com Fallback** ⚠️