MinIO in OpenShift - cniackz/public GitHub Wiki

I am totally grateful with Daniel Valdivia for finding this solution!

Objective:

To show how MinIO is installed in OpenShift, this WA was to avoid:

x509: certificate signed by unknown authority

But in latest versions this is no longer needed as we are appending the cert in go code.

Steps:

Go to: https://console.redhat.com/openshift/create/local
Download the version to your architecture:

Follow steps from: https://github.com/cniackz/public/wiki/crc and get your environment ready:

INFO All operators are available. Ensuring stability... 
INFO Operators are stable (2/3)...                
INFO Operators are stable (3/3)...                
INFO Adding crc-admin and crc-developer contexts to kubeconfig... 
Started the OpenShift cluster.

The server is accessible via web console at:
  https://console-openshift-console.apps-crc.testing

Log in as administrator:
  Username: kubeadmin
  Password: b9kmm-IyTc3-49TUK-ImHjc

Log in as user:
  Username: developer
  Password: developer

Use the 'oc' command line interface:
  $ eval $(crc oc-env)
  $ oc login -u developer https://api.crc.testing:6443

oc login -u kubeadmin https://api.crc.testing:6443

Install Operator

kubectl apply -k github.com/minio/operator/resources/\?ref\=v4.5.8

Allow permissions:

oc adm policy add-scc-to-user privileged -n minio-operator -z minio-operator
oc adm policy add-scc-to-user privileged -n minio-operator -z console-sa
oc adm policy add-scc-to-user privileged -n minio-operator -z default
oc adm policy add-scc-to-user privileged -n minio-operator -z builder
oc adm policy add-scc-to-user privileged -n minio-operator -z deployer

Reduce to one replica in the UI:

https://console-openshift-console.apps-crc.testing
Log in as administrator:
- Username: kubeadmin
- Password: b9kmm-IyTc3-49TUK-ImHjc

Create a Route for Operator and access the location:

http://my-route-minio-operator.apps-crc.testing

Get the Secret from OpenShift UI, name is something like -> minio-operator-token-wrfrp:

Copied token:

eyJhbGciOiJSUzI1NiIsImtpZCI6IlVGaXJ2cEszd0gxZEpHRl9NbWJLX3lwX0QtRjJYTFlObVhWejZhc1VibDgifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJtaW5pby1vcGVyYXRvciIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJtaW5pby1vcGVyYXRvci10b2tlbi13cmZycCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJtaW5pby1vcGVyYXRvciIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjI1ZTIwYWE4LWMyYzItNDhjNS1hYmRhLWM3ZWU1NzNiYTgyOCIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDptaW5pby1vcGVyYXRvcjptaW5pby1vcGVyYXRvciJ9.eRuTNE2DaP5XUOIIFKftlN4S8IAzOOpOh8olv8WRiLXgkJA47fBhd7xnwb3wqN9wMxCTV8sR5OTVi5QJ8BYwqHlcwnN9jVsw760ezYaIvLyvUby0dVO2aqJF5ZrzXVyMsjyNOIR4ZUITijpv4r4Hx4Fg7j78oNECWevTdH-eXkwTq7BkOdVpKLHBGPLHNrQtw5Ul8xDs1thD-o5lVUUNPCaZAJkaW-CM9JEya0HE8dCZOJRNoLR9OPEuYBo8qGg7o1cg3yRnLDRBVmonijXz8kzAYAs_eaNyMYZFptucsBZTjwddsT5OYlUXj21_olBJB8GkTr3rWEnQc30XESiaV3jrBkm0ZALyaDbscP7rofRaYD8oserjjLvaNAoVkZKaKrOoO-qlL2Ym2USuwiJfF0jKo03nz8IawbvSpoZ7YnHD5NcF5f1iOlQAGJkAo6yEybA5oTuLvBnqfaRP3MV4WWisnUuZFqFeVIqbyQCp6oownPd0mDUqMv4BzQszn2UhZM5hQl3AoqhMYBvJC5zpEddxUAFNeCLVI-7p7FYFVm1xZKtesL868Cr6T0YuR87wRttoNgH6zmJV3n_6cPRmn2ZIg7zvnCIShJe0tDjhltlDFqTcma-fE0TetEiAHv17KyAy68_L9qmM306p4goCMPMRm1Sk4f59G5mi_H2T3E4

Deploy a Tenant:

oc apply -k ~/operator/examples/kustomization/tenant-lite
oc create serviceaccount minio-operator -n tenant-lite
oc adm policy add-scc-to-user privileged -n tenant-lite -z minio-operator
oc adm policy add-scc-to-user privileged -n tenant-lite -z builder
oc adm policy add-scc-to-user privileged -n tenant-lite -z deployer
oc adm policy add-scc-to-user privileged -n tenant-lite -z default

Disable prometheus:

Change StorageClassName to crc-csi-hostpath-provisioner because that is the one is installed in Local Environment or use DirectPV:

spec.pools.volumeClaimTemplate.spec.storageClassName

spec:
  pools:
  - affinity:
      podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
            - key: v1.min.io/tenant
              operator: In
              values:
              - my-temporal-tenant
            - key: v1.min.io/pool
              operator: In
              values:
              - pool-0
          topologyKey: kubernetes.io/hostname
    name: pool-0
    resources: {}
    servers: 4
    volumeClaimTemplate:
      metadata:
        name: data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 1 Gi
        storageClassName: crc-csi-hostpath-provisioner <---- HERE!

Run this line to get the signers:

oc get secret csr-signer -n openshift-kube-controller-manager-operator -o template='{{ index .data "tls.crt"}}' | base64 -d

Expected output similar to:

Cesars-MacBook-Pro:~ cniackz$ oc get secret csr-signer -n openshift-kube-controller-manager-operator -o template='{{ index .data "tls.crt"}}' | base64 -d
-----BEGIN CERTIFICATE-----
MIIDXTCCAkWgAwIBAgIIAVNdvJOhCrQwDQYJKoZIhvcNAQELBQAwUjFQME4GA1UE
AwxHb3BlbnNoaWZ0LWt1YmUtY29udHJvbGxlci1tYW5hZ2VyLW9wZXJhdG9yX2Nz
ci1zaWduZXItc2lnbmVyQDE2NzQwMjYyNzQwHhcNMjMwMjExMTIzODU0WhcNMjMw
MzEzMTIzODU1WjAmMSQwIgYDVQQDDBtrdWJlLWNzci1zaWduZXJfQDE2NzYxMTkx
MzUwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDGXNzhqnClOHlE4tyH
8q0RBs6cCwPfyYESqCJVYnwsPXVvljnPyi3Pnyq4ds7h6FJCxw90BKoUMWROzC1g
dlIXYn4EnfedqudPNOrbgfE0+sUPx7z/TDEcP1meakDFSLC6jDFhHD9Zm5TwPVA2
PZy8trPETLHvg7bIKm3MGmyaQEfqmJu63l/Z+4bviNd8UzICJ6tgoey/kMFjzb5p
CeMxlVlJIehiVvCZza8nnyFwYGPqCxhdqGWnMsbFoRqSJMFTNTVMtD3Q1DFZABlN
Bt8J5lnDbPA1HISH8/jadLm1Cbrsa3E4D8SmtZMSLviOQnPtpbaDcIUXjkEbSpRu
H31BAgMBAAGjYzBhMA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MB0G
A1UdDgQWBBQmtOuf0AuGVipBtRvrmCxvB/j3IDAfBgNVHSMEGDAWgBTo3oG+q1dX
vsKOwjUwkjc8MzBGGDANBgkqhkiG9w0BAQsFAAOCAQEAGVvDmmRMg06bXj7t9Awg
ep+HF5uNiW1ICwtU9Co7Rj2BHzBJx4bkLrNck949SJctXfnVpoL9sb3/+uZ0Az3u
SoQS8Ar+yOB8JZ8+c8serZnp9UAIOsdZT4XRhJnrjney+POhHZHz86qlwzn4VyCN
xemMPErn07ivImD9+nrikW5t/YyBOXRw6q86KCKOd6346Fbp/uEqdwUm2cwSGl8p
JpqbPZboX1JQ0rd97iK6nDdTMtw30d4q6ePz+8n+NEGMhgk4s8/tp1xcqQQiSR1J
5BGDjinwn/TqPtkKe3Z8XF2K5/oNfsQ3cJ8KvhgSTNgXwJzGZEFDNIPBis4xjOUi
XA==
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIDiTCCAnGgAwIBAgIIX4DAQGPOuvswDQYJKoZIhvcNAQELBQAwUjFQME4GA1UE
AwxHb3BlbnNoaWZ0LWt1YmUtY29udHJvbGxlci1tYW5hZ2VyLW9wZXJhdG9yX2Nz
ci1zaWduZXItc2lnbmVyQDE2NzQwMjYyNzQwHhcNMjMwMTE4MDcxNzU0WhcNMjMw
MzE5MDcxNzU1WjBSMVAwTgYDVQQDDEdvcGVuc2hpZnQta3ViZS1jb250cm9sbGVy
LW1hbmFnZXItb3BlcmF0b3JfY3NyLXNpZ25lci1zaWduZXJAMTY3NDAyNjI3NDCC
ASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAK//l+dC9+TJXbsLwK3J7mmF
msCIzSZ8iC4X5ONnJL55lI8B58NmrpfSde2ch8qsGq7YyzVHXwXyq33QGmb2lu7k
DLjyAyRi3ynJ23ZXhodXgOjbm+PIzmmaVMagfFu2MH1z2Av3CMaps4vSn/XNmIdj
Ch4XNcvslbDSmdo+j1spLwqoIBxq2EJdIzBu+v13elZzNJlDYAYOFW3P8qaegKtA
Z9cDwZeUjLpf4h5yCUfuhY9AaizXLEbfHumK5Xy001y0dNtK7e+tE9mLXDzGhZKt
Zpyb3D8DMcHjumLnwfcbs21WbUCgK3QRydsp3XUqNwsL+LFPyirZWyS23YoPuZ0C
AwEAAaNjMGEwDgYDVR0PAQH/BAQDAgKkMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0O
BBYEFOjegb6rV1e+wo7CNTCSNzwzMEYYMB8GA1UdIwQYMBaAFOjegb6rV1e+wo7C
NTCSNzwzMEYYMA0GCSqGSIb3DQEBCwUAA4IBAQBgtbFJY6mBk1UnGyW9RGzPBjF6
+lLQtGbw8ddjq2/DPE1iMv1IVv7msb8D/8Bj1CwEP05sm3m/RHQ70hafNhuDhGu1
Ck28ILlzz0tl4babySgvNvkC+KnqkINySac/2qYYH+0GEwOi+ytHb7ggCO8AaDbc
uER6Z/5QeC/i5qtxM/8ltaH7G5sBDQZPIPUyv5ig1Y50/Sxr+tD4cJSdvSKk3l+F
+gzjoMlK03ZWnThEJR3+hmaBQZILfXXRy3qQ2dW76p5dno894AJLj1e5AITBIToG
PxeWbL9vhXca2NB4AYaThrLspbpTp56cJGX7l6sLOzcEgtewYjo1Ca10qsJg
-----END CERTIFICATE-----

In minio-operator namespace create a secret with above certificates:

apiVersion: v1
data:
  tls-a.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURYVENDQWtXZ0F3SUJBZ0lJQVZOZHZKT2hDclF3RFFZSktvWklodmNOQVFFTEJRQXdVakZRTUU0R0ExVUUKQXd4SGIzQmxibk5vYVdaMExXdDFZbVV0WTI5dWRISnZiR3hsY2kxdFlXNWhaMlZ5TFc5d1pYSmhkRzl5WDJOegpjaTF6YVdkdVpYSXRjMmxuYm1WeVFERTJOelF3TWpZeU56UXdIaGNOTWpNd01qRXhNVEl6T0RVMFdoY05Nak13Ck16RXpNVEl6T0RVMVdqQW1NU1F3SWdZRFZRUUREQnRyZFdKbExXTnpjaTF6YVdkdVpYSmZRREUyTnpZeE1Ua3gKTXpVd2dnRWlNQTBHQ1NxR1NJYjNEUUVCQVFVQUE0SUJEd0F3Z2dFS0FvSUJBUURHWE56aHFuQ2xPSGxFNHR5SAo4cTBSQnM2Y0N3UGZ5WUVTcUNKVllud3NQWFZ2bGpuUHlpM1BueXE0ZHM3aDZGSkN4dzkwQktvVU1XUk96QzFnCmRsSVhZbjRFbmZlZHF1ZFBOT3JiZ2ZFMCtzVVB4N3ovVERFY1AxbWVha0RGU0xDNmpERmhIRDlabTVUd1BWQTIKUFp5OHRyUEVUTEh2ZzdiSUttM01HbXlhUUVmcW1KdTYzbC9aKzRidmlOZDhVeklDSjZ0Z29leS9rTUZqemI1cApDZU14bFZsSkllaGlWdkNaemE4bm55RndZR1BxQ3hoZHFHV25Nc2JGb1JxU0pNRlROVFZNdEQzUTFERlpBQmxOCkJ0OEo1bG5EYlBBMUhJU0g4L2phZExtMUNicnNhM0U0RDhTbXRaTVNMdmlPUW5QdHBiYURjSVVYamtFYlNwUnUKSDMxQkFnTUJBQUdqWXpCaE1BNEdBMVVkRHdFQi93UUVBd0lDcERBUEJnTlZIUk1CQWY4RUJUQURBUUgvTUIwRwpBMVVkRGdRV0JCUW10T3VmMEF1R1ZpcEJ0UnZybUN4dkIvajNJREFmQmdOVkhTTUVHREFXZ0JUbzNvRytxMWRYCnZzS093alV3a2pjOE16QkdHREFOQmdrcWhraUc5dzBCQVFzRkFBT0NBUUVBR1Z2RG1tUk1nMDZiWGo3dDlBd2cKZXArSEY1dU5pVzFJQ3d0VTlDbzdSajJCSHpCSng0YmtMck5jazk0OVNKY3RYZm5WcG9MOXNiMy8rdVowQXozdQpTb1FTOEFyK3lPQjhKWjgrYzhzZXJabnA5VUFJT3NkWlQ0WFJoSm5yam5leStQT2hIWkh6ODZxbHd6bjRWeUNOCnhlbU1QRXJuMDdpdkltRDkrbnJpa1c1dC9ZeUJPWFJ3NnE4NktDS09kNjM0NkZicC91RXFkd1VtMmN3U0dsOHAKSnBxYlBaYm9YMUpRMHJkOTdpSzZuRGRUTXR3MzBkNHE2ZVB6KzhuK05FR01oZ2s0czgvdHAxeGNxUVFpU1IxSgo1QkdEamlud24vVHFQdGtLZTNaOFhGMks1L29OZnNRM2NKOEt2aGdTVE5nWHdKekdaRUZETklQQmlzNHhqT1VpClhBPT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQ==
  tls-b.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURpVENDQW5HZ0F3SUJBZ0lJWDREQVFHUE91dnN3RFFZSktvWklodmNOQVFFTEJRQXdVakZRTUU0R0ExVUUKQXd4SGIzQmxibk5vYVdaMExXdDFZbVV0WTI5dWRISnZiR3hsY2kxdFlXNWhaMlZ5TFc5d1pYSmhkRzl5WDJOegpjaTF6YVdkdVpYSXRjMmxuYm1WeVFERTJOelF3TWpZeU56UXdIaGNOTWpNd01URTRNRGN4TnpVMFdoY05Nak13Ck16RTVNRGN4TnpVMVdqQlNNVkF3VGdZRFZRUURERWR2Y0dWdWMyaHBablF0YTNWaVpTMWpiMjUwY205c2JHVnkKTFcxaGJtRm5aWEl0YjNCbGNtRjBiM0pmWTNOeUxYTnBaMjVsY2kxemFXZHVaWEpBTVRZM05EQXlOakkzTkRDQwpBU0l3RFFZSktvWklodmNOQVFFQkJRQURnZ0VQQURDQ0FRb0NnZ0VCQUsvL2wrZEM5K1RKWGJzTHdLM0o3bW1GCm1zQ0l6U1o4aUM0WDVPTm5KTDU1bEk4QjU4Tm1ycGZTZGUyY2g4cXNHcTdZeXpWSFh3WHlxMzNRR21iMmx1N2sKRExqeUF5UmkzeW5KMjNaWGhvZFhnT2pibStQSXptbWFWTWFnZkZ1Mk1IMXoyQXYzQ01hcHM0dlNuL1hObUlkagpDaDRYTmN2c2xiRFNtZG8rajFzcEx3cW9JQnhxMkVKZEl6QnUrdjEzZWxaek5KbERZQVlPRlczUDhxYWVnS3RBClo5Y0R3WmVVakxwZjRoNXlDVWZ1aFk5QWFpelhMRWJmSHVtSzVYeTAwMXkwZE50SzdlK3RFOW1MWER6R2haS3QKWnB5YjNEOERNY0hqdW1MbndmY2JzMjFXYlVDZ0szUVJ5ZHNwM1hVcU53c0wrTEZQeWlyWld5UzIzWW9QdVowQwpBd0VBQWFOak1HRXdEZ1lEVlIwUEFRSC9CQVFEQWdLa01BOEdBMVVkRXdFQi93UUZNQU1CQWY4d0hRWURWUjBPCkJCWUVGT2plZ2I2clYxZSt3bzdDTlRDU056d3pNRVlZTUI4R0ExVWRJd1FZTUJhQUZPamVnYjZyVjFlK3dvN0MKTlRDU056d3pNRVlZTUEwR0NTcUdTSWIzRFFFQkN3VUFBNElCQVFCZ3RiRkpZNm1CazFVbkd5VzlSR3pQQmpGNgorbExRdEdidzhkZGpxMi9EUEUxaU12MUlWdjdtc2I4RC84QmoxQ3dFUDA1c20zbS9SSFE3MGhhZk5odURoR3UxCkNrMjhJTGx6ejB0bDRiYWJ5U2d2TnZrQytLbnFrSU55U2FjLzJxWVlIKzBHRXdPaSt5dEhiN2dnQ084QWFEYmMKdUVSNlovNVFlQy9pNXF0eE0vOGx0YUg3RzVzQkRRWlBJUFV5djVpZzFZNTAvU3hyK3RENGNKU2R2U0trM2wrRgorZ3pqb01sSzAzWlduVGhFSlIzK2htYUJRWklMZlhYUnkzcVEyZFc3NnA1ZG5vODk0QUpMajFlNUFJVEJJVG9HClB4ZVdiTDl2aFhjYTJOQjRBWWFUaHJMc3BicFRwNTZjSkdYN2w2c0xPemNFZ3Rld1lqbzFDYTEwcXNKZwotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0t
kind: Secret
metadata:
  name: operator-openshift-tls
  namespace: minio-operator

oc apply -f secret.yaml

Change your Operator Deployment and mount the secret:

Read: https://kubernetes.io/docs/concepts/configuration/secret/
volumeMounts is under the container, volumes is under the spec, the idea is that container should mount the volume that contains the secret and remember the secret is nothing but the certificates. So at the end, all we want is to have /etc/ssl/certs populated with the certificates we got from OpenShift to trust the signer.

          volumeMounts:
            - name: foo
              mountPath: /etc/ssl/certs
          terminationMessagePolicy: File
          image: 'minio/operator:v4.5.8'
      serviceAccount: minio-operator
      volumes:
        - name: foo
          secret:
            secretName: operator-openshift-tls
            defaultMode: 420
      dnsPolicy: ClusterFirst

Notice users were provisioned because Operator Trust the certificate and it is initialized:

Thank you Daniel Valdivia for this finding!!!!