vm broker ‐ k8s ‐ k3s ‐ operator and tenant (kubectl minio plugin) - allanrogerr/public GitHub Wiki
This walkthru shows how to setup a minio operator using k3s
Set linger and reboot
ssh -p 20055 [email protected] -o "ServerAliveInterval=5" -o "ServerAliveCountMax=100000" -o "StrictHostKeyChecking=off"
loginctl enable-linger ubuntu
sudo reboot -h now
ssh -p 20055 [email protected] -o "ServerAliveInterval=5" -o "ServerAliveCountMax=100000" -o "StrictHostKeyChecking=off"
Install and verify k3s
sudo touch /dev/kmsg
curl -sfL https://get.k3s.io | K3S_KUBECONFIG_MODE="644" sh -s - --snapshotter=fuse-overlayfs
#sudo journalctl -f -u k3s.service
sudo systemctl status k3s
Install krew
see https://krew.sigs.k8s.io/docs/user-guide/setup/install/ for macOS/Linux > Bash or ZSH shells
(
set -x; cd "$(mktemp -d)" &&
OS="$(uname | tr '[:upper:]' '[:lower:]')" &&
ARCH="$(uname -m | sed -e 's/x86_64/amd64/' -e 's/\(arm\)\(64\)\?.*/\1\2/' -e 's/aarch64$/arm64/')" &&
KREW="krew-${OS}_${ARCH}" &&
curl -fsSLO "https://github.com/kubernetes-sigs/krew/releases/latest/download/${KREW}.tar.gz" &&
tar zxvf "${KREW}.tar.gz" &&
./"${KREW}" install krew
)
export PATH="${KREW_ROOT:-$HOME/.krew}/bin:$PATH"
Install kubectl minio
https://min.io/docs/minio/kubernetes/upstream/reference/kubectl-minio-plugin.html#installation
kubectl krew update
kubectl krew install minio
kubectl minio version
Deploy minio
kubectl minio init --console-tls
Set TLS
kubectl set env -n minio-operator deployment/minio-operator MINIO_CI_CD=on MINIO_CONSOLE_TLS_ENABLE=on
Get jwt
SA_TOKEN=$(kubectl -n minio-operator get secret console-sa-secret -o jsonpath="{.data.token}" | base64 --decode)
echo $SA_TOKEN
Create a NodePort and access the operator at https://kes-k8s-minio.lab.min.dev:30043
kubectl patch service -n minio-operator console -p '{"spec":{"ports":[{"name": "http","port": 9090,"protocol": "TCP","nodePort":31090}],"type": "NodePort"}}'
kubectl patch service -n minio-operator console -p '{"spec":{"ports":[{"name": "https","port": 9443,"protocol": "TCP","nodePort":30043}],"type": "NodePort"}}'
OPTIONAL: Fix various bugs
Can only do Serving operator at http://[::]:9090 i.e TLS cannot be enabled
kubectl patch deployment -n minio-operator console -p '{"spec":{"template":{"spec":{"volumes":[{"name": "tls-certificates", "projected": {"sources": [{"secret": {"name": "console-tls"}}]}}]}}}}'
E: 2023/11/11 04:58:46 Unable to load certs: unable to create certs CA directory at /tmp/certs/CAs: failed with mkdir /tmp/certs/CAs: read-only file system
kubectl patch deployment -n minio-operator console -p '{"spec":{"template":{"spec":{"volumes":[{"name": "cas", "emptyDir": {}}]}}}}'
kubectl patch deployment -n minio-operator console -p '{"spec":{"template":{"spec":{"containers":[{"name": "console", "image": "minio/operator:v5.0.10", "volumeMounts":[{"name": "cas", "mountPath": "/tmp/certs/CAs"}]}]}}}}'
kubectl -n tenant-lite describe pod/myminio-pool-0-2 | grep affinity Warning FailedScheduling 9m39s default-scheduler 0/1 nodes are available: 1 node(s) didn't match pod anti-affinity rules. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod..
kubectl edit tenant -n tenant-lite myminio
Change podAntiAffinity operator from In to NotIn
This patch does not work
kubectl patch tenant -n tenant-lite myminio --type='merge' -p '{"spec": {"pools": [{"affinity": {"podAntiAffinity": {"requiredDuringSchedulingIgnoredDuringExecution": [{"labelSelector": {"matchExpressions": [{"key": "v1.min.io/pool", "operator": "NotIn", "values": ["myminio"]},{"key": "v1.min.io/tenant", "operator": "NotIn", "values": ["myminio"]}]}}]}}}]}}'
Use jwt to access operator (see above echo $SA_TOKEN)
https://kes-k8s-minio.lab.min.dev:30043/login
OR
Create a port forward
kubectl -n minio-operator port-forward svc/console 9043:9443
Use jwt to access operator (see above echo $SA_TOKEN)
https://kes-k8s-minio.lab.min.dev:9043/login
Install tenant
kubectl delete ns tenant-ns
kubectl create ns tenant-ns
kubectl minio tenant create myminio --namespace tenant-ns --servers 4 --volumes 8 --capacity 16Gi
kubectl patch service -n tenant-ns myminio-console -p '{"spec":{"type": "NodePort", "ports":[{"name": "https-console", "port": 9443, "protocol": "TCP", "targetPort": 9443, "nodePort": 30090}]}}'