U1.50 Ubuntu Quick Start (QS): RabbitMq Kubernetes cluster operator. - chempkovsky/CS2WPF-and-CS2XAMARIN GitHub Wiki

Reading

We start with

Install RabbitMq cluster operator

RabbitMq cluster operator secrets

Uninstall RabbitMq cluster operator

Insufficient memory for Simple RabbitMq cluster

Create Simple RabbitMq cluster
Status of the created RabbitMq cluster
Troubleshooting
Pod memory consumption

Default storage class

Replicas

Requests and Limits

RabbitMQ Management UI

Scale the cluster

Allowed namespaces

Get a Shell to a Running Container

Scheduling pods on selected nodes

TLS

Configure TLS with tls_gen

Configure TLS with CFSSL

Configure TLS with openssl

Reading

read the article RabbitMQ Cluster Operator for Kubernetes
read the article RabbitMQ Cluster Kubernetes Operator Quickstart
read the article Installing RabbitMQ Cluster Operator in a Kubernetes Cluster

We start with

we continue to work with the Kubernetes cluster prepared in the article U1.49 Ubuntu Quick Start (QS): Kubernetes test bench on premises.

Install RabbitMq cluster operator

for u2004s01

kubectl apply -f "https://github.com/rabbitmq/cluster-operator/releases/latest/download/cluster-operator.yml"

RabbitMq cluster operator secrets

read the article Secrets
read the article Service account token Secrets
for u2004s01

yury@u2004s01:~$ kubectl get secret  -o wide -n rabbitmq-system
NAME                                    TYPE                                  DATA   AGE
default-token-64fd4                     kubernetes.io/service-account-token   3      62s
rabbitmq-cluster-operator-token-7lbfh   kubernetes.io/service-account-token   3      62s

yury@u2004s01:~$ kubectl get serviceaccounts -n rabbitmq-system
NAME                        SECRETS   AGE
default                     1         22m
rabbitmq-cluster-operator   1         22m

for u2004s01

yury@u2004s01:~$ kubectl describe secret default-token-64fd4 -n rabbitmq-system
Name:         default-token-64fd4
Namespace:    rabbitmq-system
Labels:       <none>
Annotations:  kubernetes.io/service-account.name: default
              kubernetes.io/service-account.uid: aa1946ed-f7d6-468d-b096-087b0b84d743

Type:  kubernetes.io/service-account-token

Data
====
token:      eyJhbGciOiJSUzI1NiIsImtpZC  ...  ZaA_SY1iuysLDQ
ca.crt:     1099 bytes
namespace:  15 bytes

for u2004s01

yury@u2004s01:~$ kubectl describe secret rabbitmq-cluster-operator-token-7lbfh -n rabbitmq-system
Name:         rabbitmq-cluster-operator-token-7lbfh
Namespace:    rabbitmq-system
Labels:       <none>
Annotations:  kubernetes.io/service-account.name: rabbitmq-cluster-operator
              kubernetes.io/service-account.uid: df454543-fe28-42b2-805c-17f7aa154422

Type:  kubernetes.io/service-account-token

Data
====
ca.crt:     1099 bytes
namespace:  15 bytes
token:      eyJhbGciOiJSUzI1NiIsImtpZCI6IktITG1CWC1CV  ...  334Hsvb8niuJQYYL_rQ

for u2004s01

kubectl get secret default-token-64fd4 -n rabbitmq-system -o yaml > default-token-64fd4.yaml
kubectl get secret default-token-64fd4 -n rabbitmq-system -o jsonpath="{.data['ca\.crt']}" | base64 --decode > default-token-64fd4.crt


kubectl get secret rabbitmq-cluster-operator-token-7lbfh -n rabbitmq-system -o yaml > rabbitmq-cluster-operator-token-7lbfh.yaml
kubectl get secret rabbitmq-cluster-operator-token-7lbfh -n rabbitmq-system -o jsonpath="{.data['ca\.crt']}" | base64 --decode > rabbitmq-cluster-operator-token-7lbfh.crt

openssl x509 -noout -text -in default-token-64fd4.crt
openssl x509 -noout -text -in rabbitmq-cluster-operator-token-7lbfh.crt
openssl x509 -noout -text -in /etc/kubernetes/pki/ca.crt


cat default-token-64fd4.crt
cat rabbitmq-cluster-operator-token-7lbfh.crt
cat /etc/kubernetes/pki/ca.crt

Note: default-token-64fd4.crt == rabbitmq-cluster-operator-token-7lbfh.crt == /etc/kubernetes/pki/ca.crt
read the article Certificate Management with kubeadm
for u2004s01

rm default-token-64fd4.yaml
rm default-token-64fd4.crt
rm rabbitmq-cluster-operator-token-7lbfh.yaml
rm rabbitmq-cluster-operator-token-7lbfh.crt

Uninstall RabbitMq cluster operator

for u2004s01

kubectl delete -f "https://github.com/rabbitmq/cluster-operator/releases/latest/download/cluster-operator.yml"

Insufficient memory for Simple RabbitMq cluster

Create Simple RabbitMq cluster

Note: The default storage class will be used for RabbitMq cluster. In our case it'll be second-local-path

yury@u2004s01:~$ kubectl get sc -A
NAME                          PROVISIONER                    RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
first-local-path              rancher.io/first-local-path    Delete          WaitForFirstConsumer   true                   4d
second-local-path (default)   rancher.io/second-local-path   Delete          WaitForFirstConsumer   true                   4d

for u2004s01

kubectl apply -f https://raw.githubusercontent.com/rabbitmq/cluster-operator/main/docs/examples/hello-world/rabbitmq.yaml

Status of the created RabbitMq cluster

yury@u2004s01:~$ kubectl get pvc,pod
yury@u2004s01:~$ kubectl get all

click to show the response

yury@u2004s01:~$ kubectl get pvc,pod
NAME                                                     STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS        AGE
persistentvolumeclaim/persistence-hello-world-server-0   Pending                                      second-local-path   20m

NAME                       READY   STATUS    RESTARTS   AGE
pod/hello-world-server-0   0/1     Pending   0          20m


Every 2.0s: kubectl get all                                            u2004s01: Mon Jan  3 16:13:41 2022

NAME                       READY   STATUS    RESTARTS   AGE
pod/hello-world-server-0   0/1     Pending   0          12m

NAME                        TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                        AGE
service/hello-world         ClusterIP   10.105.139.178   <none>        5672/TCP,15672/TCP,15692/TCP   12m
service/hello-world-nodes   ClusterIP   None             <none>        4369/TCP,25672/TCP             12m
service/kubernetes          ClusterIP   10.96.0.1        <none>        443/TCP                        4d5h

NAME                                  READY   AGE
statefulset.apps/hello-world-server   0/1     12m

NAME                                       ALLREPLICASREADY   RECONCILESUCCESS   AGE
rabbitmqcluster.rabbitmq.com/hello-world   False              Unknown            12m

pod/hello-world-server-0 has the Pending status. Let's get more information:
- for u2004s01

kubectl describe pod/hello-world-server-0

click to show the response

...
Events:
  Type     Reason            Age                  From               Message
  ----     ------            ----                 ----               -------
  Warning  FailedScheduling  57s (x5 over 5m18s)  default-scheduler  0/4 nodes are available: 1 Insufficient cpu, 4 Insufficient memory.

We have Insufficient cpu and Insufficient memory.

Troubleshooting

At Hyper-V side it was set for every virtual machine:
- Settings/memory/Ram = 2048
- Settings/memory/Enable Dynamic memory = Yes
- Settings/memory/Minimum ram = 2048
- Settings/memory/Maximum ram = 1048576
- Settings/Number of vrtual processors = 2
Step 1
- for u2004s01

kubectl delete -f https://raw.githubusercontent.com/rabbitmq/cluster-operator/main/docs/examples/hello-world/rabbitmq.yaml

Step 2
- for each virtual machine

sudo poweroff

Step 3
- for each virtual machine
  - At Hyper-V side it was set for every virtual machine:
    - Settings/memory/Ram = 4096
    - Settings/memory/Enable Dynamic memory = Yes
    - Settings/memory/Minimum ram = 4096
    - Settings/memory/Maximum ram = 1048576
    - Settings/Number of vrtual processors = 4
Step 4
- Start all virtual machines
Step 5
- for u2004s01

kubectl apply -f https://raw.githubusercontent.com/rabbitmq/cluster-operator/main/docs/examples/hello-world/rabbitmq.yaml

here is a result

yury@u2004s01:~$ kubectl get pods
NAME                   READY   STATUS    RESTARTS   AGE
hello-world-server-0   1/1     Running   0          117s

Pod memory consumption

We set
- Enable Dynamic memory=Yes
- Settings/memory/Minimum ram=4096
for u2004s01

kubectl describe node/u2004s02

Here is a result
- It looks like Kuberenetes is not using Hyper-V dynamic memory

...
Capacity:
  cpu:                4
  ephemeral-storage:  64757356Ki
  hugepages-2Mi:      0
  memory:             4016440Ki
...

Default storage class

We're going to install hello-world without the default storage class
for u2004s01

kubectl delete -f https://raw.githubusercontent.com/rabbitmq/cluster-operator/main/docs/examples/hello-world/rabbitmq.yaml
kubectl patch storageclass second-local-path -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'
kubectl apply -f https://raw.githubusercontent.com/rabbitmq/cluster-operator/main/docs/examples/hello-world/rabbitmq.yaml

The result is as expected

yury@u2004s01:~$ kubectl get pods -o wide
NAME                   READY   STATUS    RESTARTS   AGE   IP       NODE     NOMINATED NODE   READINESS GATES
hello-world-server-0   0/1     Pending   0          20s   <none>   <none>   <none>           <none>

yury@u2004s01:~$ kubectl describe pod/hello-world-server-0
...
Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  74s   default-scheduler  0/4 nodes are available: 4 pod has unbound immediate PersistentVolumeClaims.

for u2004s01

kubectl delete -f https://raw.githubusercontent.com/rabbitmq/cluster-operator/main/docs/examples/hello-world/rabbitmq.yaml

kubectl apply -f- <<EOF
apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  name: hello-world
spec:
  persistence:
    storageClassName: second-local-path
EOF

kubectl get pods -o wide

here is a result

yury@u2004s01:~$ kubectl get pods -o wide
NAME                   READY   STATUS    RESTARTS   AGE   IP              NODE       NOMINATED NODE   READINESS GATES
hello-world-server-0   0/1     Running   0          18s   10.32.121.143   u2004s04   <none>           <none>

Replicas

for u2004s01

kubectl delete -f- <<EOF
apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  name: hello-world
spec:
  persistence:
    storageClassName: second-local-path
EOF


kubectl apply -f- <<EOF
apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  name: hello-world
spec:
  replicas: 2
  persistence:
    storageClassName: second-local-path
EOF

here is a result

yury@u2004s01:~$ kubectl get pods -o wide
NAME                   READY   STATUS    RESTARTS   AGE   IP              NODE       NOMINATED NODE   READINESS GATES
hello-world-server-0   0/1     Running   0          62s   10.32.105.10    u2004s03   <none>           <none>
hello-world-server-1   1/1     Running   0          61s   10.32.121.146   u2004s04   <none>           <none>

Requests and Limits

for u2004s01

kubectl delete -f- <<EOF
apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  name: hello-world
spec:
  replicas: 2
  persistence:
    storageClassName: second-local-path
EOF



kubectl apply -f- <<EOF
apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  name: hello-world
spec:
  replicas: 2
  persistence:
    storageClassName: second-local-path
    storage: 5Gi
  resources:
    requests:
      cpu: 100m
      memory: 2Gi
    limits:
      cpu: 100m
      memory: 2Gi
EOF

here is a result

yury@u2004s01:~$ kubectl get pods -o wide
NAME                   READY   STATUS    RESTARTS   AGE     IP              NODE       NOMINATED NODE   READINESS GATES
hello-world-server-0   1/1     Running   0          7m46s   10.32.105.16    u2004s03   <none>           <none>
hello-world-server-1   1/1     Running   0          7m46s   10.32.121.152   u2004s04   <none>           <none>

getting events:

yury@u2004s01:~$ kubectl get event --namespace default --field-selector involvedObject.name=hello-world-server-0
LAST SEEN   TYPE      REASON             OBJECT                     MESSAGE
...
y13m        Warning   Unhealthy          pod/hello-world-server-0   Readiness probe failed: dial tcp 10.32.105.16:5672: connect: connection refused

yury@u2004s01:~$ kubectl get event --namespace default --field-selector involvedObject.name=hello-world-server-1
LAST SEEN   TYPE      REASON      OBJECT                     MESSAGE
...
14m         Warning   Unhealthy   pod/hello-world-server-1   Readiness probe failed: dial tcp 10.32.121.152:5672: connect: connection refused

RabbitMQ Management UI

read the article Find Your RabbitmqCluster Service Name and Admin Credentials
To retrieve the Secret name
- for u2004s01

yury@u2004s01:~$ kubectl get rabbitmqcluster hello-world -ojsonpath='{.status.defaultUser.secretReference.name}'
hello-world-default-user

To retrieve the username
- for u2004s01

yury@u2004s01:~$ kubectl -n default get secret hello-world-default-user -o jsonpath="{.data.username}" | base64 --decode
default_user_MN-6YXVLFsqYrvfyhdJ

To retrieve the password
- for u2004s01

yury@u2004s01:~$ kubectl -n default get secret hello-world-default-user -o jsonpath="{.data.password}" | base64 --decode
uS5kFxsjKJbKOY8lvvkTY2pXv4O-Scig

To start proxy
- for u2004s01

kubectl port-forward --address 0.0.0.0 svc/hello-world 15672

in browser go to the address below
- enter username=default_user_MN-6YXVLFsqYrvfyhdJ and password=uS5kFxsjKJbKOY8lvvkTY2pXv4O-Scig

http://192.168.100.61:15672/

Scale the cluster

for u2004s01

yury@u2004s01:~$ kubectl scale statefulset.apps/hello-world-server --replicas 3
statefulset.apps/hello-world-server scaled

yury@u2004s01:~$ kubectl get pods -o wide
NAME                   READY   STATUS    RESTARTS   AGE     IP              NODE       NOMINATED NODE   READINESS GATES
hello-world-server-0   1/1     Running   0          94m     10.32.105.20    u2004s03   <none>           <none>
hello-world-server-1   1/1     Running   0          94m     10.32.121.156   u2004s04   <none>           <none>
hello-world-server-2   0/1     Running   0          4m44s   10.32.26.206    u2004s01   <none>           <none>

yury@u2004s01:~$ kubectl get pods -o wide
NAME                   READY   STATUS    RESTARTS   AGE     IP              NODE       NOMINATED NODE   READINESS GATES
hello-world-server-0   1/1     Running   0          95m     10.32.105.20    u2004s03   <none>           <none>
hello-world-server-1   1/1     Running   0          95m     10.32.121.156   u2004s04   <none>           <none>
hello-world-server-2   1/1     Running   0          5m31s   10.32.26.206    u2004s01   <none>           <none>

yury@u2004s01:~$ kubectl scale statefulset.apps/hello-world-server --replicas 2
statefulset.apps/hello-world-server scaled

yury@u2004s01:~$ kubectl get pods -o wide
NAME                   READY   STATUS        RESTARTS   AGE    IP              NODE       NOMINATED NODE   READINESS GATES
hello-world-server-0   1/1     Running       0          97m    10.32.105.20    u2004s03   <none>           <none>
hello-world-server-1   1/1     Running       0          97m    10.32.121.156   u2004s04   <none>           <none>
hello-world-server-2   1/1     Terminating   0          8m1s   10.32.26.206    u2004s01   <none>           <none>
yury@u2004s01:~$ kubectl get pods -o wide
NAME                   READY   STATUS    RESTARTS   AGE    IP              NODE       NOMINATED NODE   READINESS GATES
hello-world-server-0   1/1     Running   0          111m   10.32.105.20    u2004s03   <none>           <none>
hello-world-server-1   1/1     Running   0          111m   10.32.121.156   u2004s04   <none>           <none>

Allowed namespaces

read the article Scope across multiple namespaces
for u2004s01

kubectl delete -f- <<EOF
apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  name: hello-world
spec:
  replicas: 2
  persistence:
    storageClassName: second-local-path
    storage: 5Gi
  resources:
    requests:
      cpu: 100m
      memory: 2Gi
    limits:
      cpu: 100m
      memory: 2Gi
EOF


kubectl create namespace hello-world-cluster-namespace

kubectl apply -f- <<EOF
apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  name: hello-world
  namespace: hello-world-cluster-namespace
  annotations:
    rabbitmq.com/topology-allowed-namespaces: "default,hello-world-queue1-namespace,hello-world-queue2-namespace"
spec:
  replicas: 2
  persistence:
    storageClassName: second-local-path
    storage: 5Gi
  resources:
    requests:
      cpu: 100m
      memory: 2Gi
    limits:
      cpu: 100m
      memory: 2Gi
EOF

kubectl get pods -o wide -n hello-world-cluster-namespace
NAME                   READY   STATUS    RESTARTS   AGE    IP              NODE       NOMINATED NODE   READINESS GATES
hello-world-server-0   1/1     Running   0          6m7s   10.32.26.209    u2004s01   <none>           <none>
hello-world-server-1   1/1     Running   0          6m7s   10.32.121.159   u2004s04   <none>           <none>

Get a Shell to a Running Container

read the article Get a Shell to a Running Container
for u2004s01

yury@u2004s01:~$ kubectl exec --stdin --tty hello-world-server-0 -n hello-world-cluster-namespace -- /bin/bash
Defaulted container "rabbitmq" out of: rabbitmq, setup-container (init)

rabbitmq@hello-world-server-0:/$ rabbitmqctl list_users
Listing users ...
user    tags
default_user_UbUl4WQWUHjjcLh6fOl        [administrator]

rabbitmq@hello-world-server-0:/$ rabbitmqctl add_user admin adminpassword
Adding user "admin" ...
Done. Don't forget to grant the user permissions to some virtual hosts! See 'rabbitmqctl help set_permissions' to learn more.

rabbitmq@hello-world-server-0:/$ rabbitmqctl set_user_tags admin administrator
Setting tags for user "admin" to [administrator] ...

rabbitmq@hello-world-server-0:/$ rabbitmqctl set_permissions -p "/" "admin" ".*" ".*" ".*"
Setting permissions for user "admin" in vhost "/" ...

rabbitmq@hello-world-server-0:/$ rabbitmqctl set_topic_permissions -p "/" "admin" ".*" ".*" ".*"
Setting topic permissions on ".*" for user "admin" in vhost "/" ...

rabbitmq@hello-world-server-0:/$ rabbitmqctl list_users
Listing users ...
user    tags
admin   [administrator]
default_user_UbUl4WQWUHjjcLh6fOl        [administrator]

rabbitmq@hello-world-server-0:/$ rabbitmqctl delete_user "admin"
Deleting user "admin" ...

rabbitmq@hello-world-server-0:/$ rabbitmqctl list_users
Listing users ...
user    tags
default_user_UbUl4WQWUHjjcLh6fOl        [administrator]

rabbitmq@hello-world-server-0:/$ exit
exit

for u2004s01

yury@u2004s01:~$ kubectl exec hello-world-server-0 -n hello-world-cluster-namespace -- rabbitmqctl list_users
Defaulted container "rabbitmq" out of: rabbitmq, setup-container (init)
Listing users ...
user    tags
default_user_UbUl4WQWUHjjcLh6fOl        [administrator]
yury@u2004s01:~$

Scheduling pods on selected nodes

read the article Affinity and Anti-affinity Rules
for u2004s01

kubectl delete -f- <<EOF
apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  name: hello-world
  namespace: hello-world-cluster-namespace
  annotations:
    rabbitmq.com/topology-allowed-namespaces: "default,hello-world-queue1-namespace,hello-world-queue2-namespace"
spec:
  replicas: 2
  persistence:
    storageClassName: second-local-path
    storage: 5Gi
  resources:
    requests:
      cpu: 100m
      memory: 2Gi
    limits:
      cpu: 100m
      memory: 2Gi
EOF


kubectl apply -f- <<EOF
apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  name: hello-world
  namespace: hello-world-cluster-namespace
  annotations:
    rabbitmq.com/topology-allowed-namespaces: "default,hello-world-queue1-namespace,hello-world-queue2-namespace"
spec:
  replicas: 2
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: kubernetes.io/hostname
            operator: In
            values:
            - u2004s02
            - u2004s03
  persistence:
    storageClassName: second-local-path
    storage: 5Gi
  resources:
    requests:
      cpu: 100m
      memory: 2Gi
    limits:
      cpu: 100m
      memory: 2Gi
EOF

yury@u2004s01:~$ kubectl get pods -n hello-world-cluster-namespace -o wide
NAME                   READY   STATUS    RESTARTS   AGE    IP             NODE       NOMINATED NODE   READINESS GATES
hello-world-server-0   1/1     Running   0          5m5s   10.32.105.23   u2004s03   <none>           <none>
hello-world-server-1   1/1     Running   0          5m5s   10.32.27.202   u2004s02   <none>           <none>

the result: three pods are scheduled on two nodes

yury@u2004s01:~$ kubectl scale statefulset.apps/hello-world-server -n hello-world-cluster-namespace --replicas 3
statefulset.apps/hello-world-server scaled

yury@u2004s01:~$ kubectl get pods -n hello-world-cluster-namespace -o wide
NAME                   READY   STATUS    RESTARTS        AGE     IP             NODE       NOMINATED NODE   READINESS GATES
hello-world-server-0   1/1     Running   1 (9m6s ago)    24m     10.32.105.25   u2004s03   <none>           <none>
hello-world-server-1   1/1     Running   1 (8m18s ago)   24m     10.32.27.205   u2004s02   <none>           <none>
hello-world-server-2   1/1     Running   0               6m18s   10.32.105.27   u2004s03   <none>           <none>

yury@u2004s01:~$ kubectl scale statefulset.apps/hello-world-server -n hello-world-cluster-namespace --replicas 2
statefulset.apps/hello-world-server scaled

yury@u2004s01:~$ kubectl get pods -n hello-world-cluster-namespace -o wide
NAME                   READY   STATUS    RESTARTS      AGE   IP             NODE       NOMINATED NODE   READINESS GATES
hello-world-server-0   1/1     Running   1 (15m ago)   31m   10.32.105.25   u2004s03   <none>           <none>
hello-world-server-1   1/1     Running   1 (14m ago)   31m   10.32.27.205   u2004s02   <none>           <none>

TLS

Kubernetes Related Articles

read the article Configure TLS
read the article Mutual TLS encryption between clients and RabbitMQ
read the article Kubernetes Certificates
- easyrsa
- openssl
- cfssl
read the article Certificate Management with kubeadm
- Set up a signer
  - Important note: The Kubernetes Certificate Authority does not work out of the box. You can configure an external signer such as cert-manager, or you can use the built-in signer.
read the article Certificate Signing Requests

RabbitMq Related Articles

read the article U1.18 Ubuntu Quick Start (QS): RabbitMq Enable HTTPS for Management UI
- read the article HTTPS
read the article U1.19 Ubuntu Quick Start (QS): RabbitMq Enable TLS for Inter node Communication and CLI tools
read the article Securing Cluster (Inter-node) and CLI Tool Communication with TLS (SSL)
- read the article Strategy One
- read the article Strategy Two

Reminder for RabbitMQ on bare metal

The management plugin can be configured to use HTTPS
- To Enable HTTPS rabbitmq.conf should contains the lines

management.tcp.port       = none

management.ssl.port       = 15671
management.ssl.cacertfile = /var/lib/rabbitmq/certs/ca_certificate.pem
management.ssl.certfile   = /var/lib/rabbitmq/certs/server_certificate.pem
management.ssl.keyfile    = /var/lib/rabbitmq/certs/server_key.pem
management.ssl.password   = bunnies

To Enable TLS for Inter node Communication and CLI tools
- rabbitmq.conf should contains the lines

listeners.tcp          = none
# listeners.ssl.1        = 5671
listeners.ssl.default  = 5671
ssl_options.versions   = tlsv1.3
ssl_options.cacertfile = /var/lib/rabbitmq/certs/ca_certificate.pem
ssl_options.certfile   = /var/lib/rabbitmq/certs/server_certificate.pem
ssl_options.keyfile    = /var/lib/rabbitmq/certs/server_key.pem
ssl_options.password   = bunnies
ssl_options.verify     = verify_peer
ssl_options.fail_if_no_peer_cert = true

Define names

for K8s cluster domain name read the article Installing Control Plane Node for u2004s01
- K8s cluster domain name = testcluster.local
for RabbitMQ cluster name and RabbitMQ namespace go to Scheduling pods on selected nodes
- RabbitMQ cluster name = hello-world
- RabbitMQ cluster namespace = hello-world-cluster-namespace
- RabbitMQ cluster node names
  - hello-world-server-0
  - hello-world-server-1
according to TLS encrypting traffic between clients and RabbitMQ
- we need to define Subject Alternative Name (SAN)
  - <RabbitMQ cluster name>.<namespace>.svc.<K8s cluster domain name> =
    - hello-world.hello-world-cluster-namespace.svc.testcluster.local
  - *.<RabbitMQ cluster name>-nodes.<namespace>.svc.<K8s cluster domain name> =
    - hello-world-server-0.hello-world-nodes.hello-world-cluster-namespace.svc.testcluster.local
  - *.<RabbitMQ cluster name>-nodes.<namespace>.svc.<K8s cluster domain name> =
    - hello-world-server-1.hello-world-nodes.hello-world-cluster-namespace.svc.testcluster.local
- Article TLS encrypting traffic between clients and RabbitMQ notes
  - Depending on the service type used (spec.service.type), further SAN attributes may be required. For example, if using service type NodePort, the SAN must include the external IP address of each Kubernetes node.

Configure TLS with tls_gen

Note: it didn’t work
read the article U1.16 Ubuntu Quick Start (QS): RabbitMq tls_gen

Install tls_gen

for u2004s01
- follow the instruction of the article U1.16 Ubuntu Quick Start (QS): RabbitMq tls_gen
  - do not go through the last step Using tls-gen's Basic Profile

modify openssl.cnf

under the steps of U1.16 Ubuntu Quick Start (QS): RabbitMq tls_gen we have created the folder
- tls-gen
run one of the commands:

nano ~/tls-gen/basic/openssl.cnf

in the file basic openssl.cnf modify [ server_alt_names ] section

[ server_alt_names ]
# DNS.1 = $common_name
# DNS.2 = $server_alt_name
# DNS.3 = localhost
DNS.1 = localhost
DNS.2 = hello-world.hello-world-cluster-namespace.svc.testcluster.local
DNS.3 = hello-world-server-0.hello-world-nodes.hello-world-cluster-namespace.svc.testcluster.local
DNS.4 = hello-world-server-1.hello-world-nodes.hello-world-cluster-namespace.svc.testcluster.local

Generate files

for u2004s01

yury@u2004s01:~$ cd tls-gen/basic

yury@u2004s01:~/tls-gen/basic$ make PASSWORD=bunnies CN=hello-world.hello-world-cluster-namespace.svc.testcluster.local

yury@u2004s01:~/tls-gen/basic$ ls -al result
total 40
drwxrwxr-x 2 yury yury 4096 Jan  5 12:08 .
drwxrwxr-x 6 yury yury 4096 Jan  5 12:08 ..
-rw-rw-r-- 1 yury yury 1208 Jan  5 12:08 ca_certificate.pem
-rw------- 1 yury yury 1854 Jan  5 12:08 ca_key.pem
-rw-rw-r-- 1 yury yury 1310 Jan  5 12:08 client_certificate.pem
-rw------- 1 yury yury 2525 Jan  5 12:08 client_key.p12
-rw------- 1 yury yury 1874 Jan  5 12:08 client_key.pem
-rw-rw-r-- 1 yury yury 1643 Jan  5 12:08 server_certificate.pem
-rw------- 1 yury yury 2773 Jan  5 12:08 server_key.p12
-rw------- 1 yury yury 1874 Jan  5 12:08 server_key.pem

Generated files

for u2004s01

yury@u2004s01:~$ cd ~/tls-gen/basic/result
yury@u2004s01:~/tls-gen/basic/result$ openssl rsa -noout -text -in ca_key.pem
Enter pass phrase for ca_key.pem:bunnies
yury@u2004s01:~/tls-gen/basic/result$ openssl rsa -noout -text -in client_key.pem
Enter pass phrase for client_key.pem:bunnies
yury@u2004s01:~/tls-gen/basic/result$ openssl rsa -noout -text -in client_key.pem
Enter pass phrase for client_key.pem:bunnies
yury@u2004s01:~/tls-gen/basic/result$ openssl rsa -noout -text -in  server_key.pem
Enter pass phrase for server_key.pem:bunnies

yury@u2004s01:~/tls-gen/basic/result$ openssl x509 -noout -text -in ca_certificate.pem
yury@u2004s01:~/tls-gen/basic/result$ openssl x509 -noout -text -in client_certificate.pem
yury@u2004s01:~/tls-gen/basic/result$ openssl x509 -noout -text -in server_certificate.pem
yury@u2004s01:~/tls-gen/basic/result$ ls -al result

Note: The last four commands above show
- client_certificate.pem and server_certificate.pem were signed with ca_certificate.pem
- there is no Certificate signing request-file

Configure TLS with CFSSL

read the article Manage TLS Certificates in a Cluster

Install CFSSL

read the article How to install CFSSL
for u2004s01

sudo apt-get update -y
sudo apt-get install -y golang-cfssl

Generate csr and pem files

for u2004s01

mkdir tmp
cd tmp
cat <<EOF | cfssl genkey - | cfssljson -bare server
{
  "hosts": [
    "hello-world.hello-world-cluster-namespace.svc.testcluster.local",
    "hello-world-server-0.hello-world-nodes.hello-world-cluster-namespace.svc.testcluster.local",
    "hello-world-server-1.hello-world-nodes.hello-world-cluster-namespace.svc.testcluster.local"
  ],
  "CN": "system:node:hello-world-server-0.hello-world-nodes.hello-world-cluster-namespace.svc.testcluster.local",
  "key": {
    "algo": "ecdsa",
    "size": 256
  },
  "names": [
    {
      "O": "system:nodes"
    }
  ]
}
EOF

yury@u2004s01:~/tmp$ ls -al
-rw-r--r-- 1 yury yury  907 Jan  5 15:22 server.csr
-rw------- 1 yury yury  227 Jan  5 15:22 server-key.pem

Create Signing Request

for u2004s01

yury@u2004s01:~/tmp$ cat <<EOF > request.yaml
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
  name: hello-world.hello-world-cluster-namespace
spec:
  request: $(cat server.csr | base64 | tr -d '\n')
  signerName: kubernetes.io/kubelet-serving
  usages:
  - digital signature
  - key encipherment
  - server auth
EOF

yury@u2004s01:~/tmp$ kubectl apply -f request.yaml
certificatesigningrequest.certificates.k8s.io/hello-world.hello-world-cluster-namespace created

yury@u2004s01:~/tmp$ kubectl describe csr hello-world.hello-world-cluster-namespace

yury@u2004s01:~/tmp$ kubectl certificate approve hello-world.hello-world-cluster-namespace
certificatesigningrequest.certificates.k8s.io/hello-world.hello-world-cluster-namespace approved

yury@u2004s01:~/tmp$ kubectl get csr
NAME                                        AGE     SIGNERNAME                      REQUESTOR          REQUESTEDDURATION   CONDITION
hello-world.hello-world-cluster-namespace   2m58s   kubernetes.io/kubelet-serving   kubernetes-admin   <none>              Approved,Issued

yury@u2004s01:~/tmp$ kubectl get csr hello-world.hello-world-cluster-namespace -o jsonpath='{.status.certificate}' | base64 --decode > server.crt

yury@u2004s01:~/tmp$ ls -al
-rw-rw-r-- 1 yury yury 1470 Jan  5 15:35 request.yaml
-rw-rw-r-- 1 yury yury 1363 Jan  5 15:45 server.crt
-rw-r--r-- 1 yury yury  907 Jan  5 15:22 server.csr
-rw------- 1 yury yury  227 Jan  5 15:22 server-key.pem

view server.crt
- for u2004s01

yury@u2004s01:~/tmp$ openssl x509 -noout -text -in server.crt
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            e3:b3:cb:8f:29:ef:5d:7a:90:a8:fb:d3:16:9d:44:d1
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = kubernetes
        Validity
            Not Before: Jan  5 15:32:34 2022 GMT
            Not After : Jan  5 15:32:34 2023 GMT
        Subject: O = system:nodes, CN = system:node:hello-world-server-0.hello-world-nodes.hello-world-cluster-namespace.svc.testcluster.local
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (256 bit)
                pub:
                    04:4f:e0:7c:66:a0:07:f3:f5:79:36:a6:ee:7a:e1:
                    86:6c:06:aa:7a:88:1f:71:64:da:47:bc:62:ba:ae:
                    2d:3b:fb:d4:f8:c6:93:99:e2:2e:fd:2f:40:00:1e:
                    14:32:47:6f:41:ab:de:a8:d5:d9:58:ff:4f:0e:bf:
                    ed:28:a6:d4:1a
                ASN1 OID: prime256v1
                NIST CURVE: P-256
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Server Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Authority Key Identifier:
                keyid:3C:61:48:36:9A:E7:3A:1B:02:11:66:CA:BC:D2:C3:7D:FE:64:0E:B6

            X509v3 Subject Alternative Name:
                DNS:hello-world.hello-world-cluster-namespace.svc.testcluster.local, DNS:hello-world-server-0.hello-world-nodes.hello-world-cluster-namespace.svc.testcluster.local, DNS:hello-world-server-1.hello-world-nodes.hello-world-cluster-namespace.svc.testcluster.local
    Signature Algorithm: sha256WithRSAEncryption
             ...

compare
- X509v3 Authority Key Identifier: keyid:3C:61:48:36:9A:E7:3A:1B:02:11:66:CA:BC:D2:C3:7D:FE:64:0E:B6 of the server.crt-file above
- and
- X509v3 Subject Key Identifier: 3C:61:48:36:9A:E7:3A:1B:02:11:66:CA:BC:D2:C3:7D:FE:64:0E:B6 of the /etc/kubernetes/pki/ca.crt-file
it means /etc/kubernetes/pki/ca.crt is used to sign server.crt-file above.

Create a Kubernetes secret

for u2004s01

yury@u2004s01:~/tmp$ kubectl create secret tls hello-world-tls-secret --cert=server.crt --key=server-key.pem
secret/hello-world-tls-secret created

yury@u2004s01:~/tmp$ kubectl get secret
NAME                     TYPE                                  DATA   AGE
default-token-4bwtk      kubernetes.io/service-account-token   3      6d5h
hello-world-tls-secret   kubernetes.io/tls                     2      2m4s

Recreate RabbitMQ cluster

for u2004s01

kubectl delete -f- <<EOF
apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  name: hello-world
  namespace: hello-world-cluster-namespace
  annotations:
    rabbitmq.com/topology-allowed-namespaces: "default,hello-world-queue1-namespace,hello-world-queue2-namespace"
spec:
  replicas: 2
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: kubernetes.io/hostname
            operator: In
            values:
            - u2004s02
            - u2004s03
  persistence:
    storageClassName: second-local-path
    storage: 5Gi
  resources:
    requests:
      cpu: 100m
      memory: 2Gi
    limits:
      cpu: 100m
      memory: 2Gi
EOF

kubectl apply -f- <<EOF
apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  name: hello-world
  namespace: hello-world-cluster-namespace
  annotations:
    rabbitmq.com/topology-allowed-namespaces: "default,hello-world-queue1-namespace,hello-world-queue2-namespace"
spec:
  replicas: 2
  tls:
    secretName: hello-world-tls-secret
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: kubernetes.io/hostname
            operator: In
            values:
            - u2004s02
            - u2004s03
  persistence:
    storageClassName: second-local-path
    storage: 5Gi
  resources:
    requests:
      cpu: 100m
      memory: 2Gi
    limits:
      cpu: 100m
      memory: 2Gi
EOF

Troubleshooting Step 1
- get rabbitmq-cluster-operator logs

yury@u2004s01:~$ kubectl get pods -n hello-world-cluster-namespace -o wide
No resources found in hello-world-cluster-namespace namespace.

yury@u2004s01:~$ kubectl -n rabbitmq-system logs -l app.kubernetes.io/name=rabbitmq-cluster-operator
{"level":"error","ts":1641400151.1549711,"logger":"controller-runtime.manager.controller.rabbitmqcluster","msg":"Reconciler error","reconciler group":"rabbitmq.com","reconciler kind":"RabbitmqCluster","name":"hello-world","namespace":"hello-world-cluster-namespace","error":"Secret \"hello-world-tls-secret\" not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:253\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:214"}

Troubleshooting Step 2
- we should change the namespace for the secret

yury@u2004s01:~$ kubectl delete secret  hello-world-tls-secret
yury@u2004s01:~$ cd tmp
yury@u2004s01:~/tmp$ kubectl create secret tls hello-world-tls-secret -n hello-world-cluster-namespace  --cert=server.crt --key=server-key.pem

kubectl delete -f- <<EOF
apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  name: hello-world
  namespace: hello-world-cluster-namespace
  annotations:
    rabbitmq.com/topology-allowed-namespaces: "default,hello-world-queue1-namespace,hello-world-queue2-namespace"
spec:
  replicas: 2
  tls:
    secretName: hello-world-tls-secret
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: kubernetes.io/hostname
            operator: In
            values:
            - u2004s02
            - u2004s03
  persistence:
    storageClassName: second-local-path
    storage: 5Gi
  resources:
    requests:
      cpu: 100m
      memory: 2Gi
    limits:
      cpu: 100m
      memory: 2Gi
EOF

kubectl apply -f- <<EOF
apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  name: hello-world
  namespace: hello-world-cluster-namespace
  annotations:
    rabbitmq.com/topology-allowed-namespaces: "default,hello-world-queue1-namespace,hello-world-queue2-namespace"
spec:
  replicas: 2
  tls:
    secretName: hello-world-tls-secret
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: kubernetes.io/hostname
            operator: In
            values:
            - u2004s02
            - u2004s03
  persistence:
    storageClassName: second-local-path
    storage: 5Gi
  resources:
    requests:
      cpu: 100m
      memory: 2Gi
    limits:
      cpu: 100m
      memory: 2Gi
EOF

NAME                   READY   STATUS    RESTARTS   AGE     IP             NODE       NOMINATED NODE   READINESS GATES
hello-world-server-0   1/1     Running   0          5m19s   10.32.105.39   u2004s03   <none>           <none>
hello-world-server-1   1/1     Running   0          5m18s   10.32.27.213   u2004s02   <none>           <none>

Configure TLS with openssl

Step 1: create a private key

without password

openssl genrsa -out server-key.key 2048

or with password

openssl genrsa -des3 -out server-key.key 2048

Step 2: create Certificate signing request (CSR)

openssl req -new -key server-key.key -out server-key.csr

...
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:Michigan
Locality Name (eg, city) []:Ann Arbor
Organization Name (eg, company) [Internet Widgits Pty Ltd]:MyCompany
Organizational Unit Name (eg, section) []:MyDep
Common Name (e.g. server FQDN or YOUR name) []:hello-world.hello-world-cluster-namespace.svc.testcluster.local
Email Address []:[email protected]
...

The downside of the command above is that we cannot add two additional DNS names:
- hello-world-server-0.hello-world-nodes.hello-world-cluster-namespace.svc.testcluster.local
- hello-world-server-1.hello-world-nodes.hello-world-cluster-namespace.svc.testcluster.local

to solve this problem

create the file

nano openssl-req.conf

with the content

[req]
distinguished_name = req_distinguished_name
req_extensions = v3_req
prompt = no
[req_distinguished_name]
C = US
ST = Michigan
L = Ann Arbor
O = MyCompany
OU = MyDep
CN = hello-world.hello-world-cluster-namespace.svc.testcluster.local
[v3_req]
keyUsage = keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
[alt_names]
DNS.1 = hello-world.hello-world-cluster-namespace.svc.testcluster.local
DNS.2 = hello-world-server-0.hello-world-nodes.hello-world-cluster-namespace.svc.testcluster.local
DNS.3 = hello-world-server-1.hello-world-nodes.hello-world-cluster-namespace.svc.testcluster.local

run the command

openssl req -new -key server-key.key -config openssl-req.conf -out server-key.csr

here is a result

yury@u2004s01:~/ch$ openssl req -noout -text -in server-key.csr
Certificate Request:
    Data:
        Version: 1 (0x0)
        Subject: C = US, ST = Michigan, L = Ann Arbor, O = MyCompany, OU = MyDep, CN = hello-world.hello-world-cluster-namespace.svc.testcluster.local
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Modulus:
                    00:b6:88:17:03:4f:6f:e4:62:76:01:37:c7:45:34:
                    ...
                Exponent: 65537 (0x10001)
        Attributes:
        Requested Extensions:
            X509v3 Key Usage:
                Key Encipherment, Data Encipherment
            X509v3 Extended Key Usage:
                TLS Web Server Authentication
            X509v3 Subject Alternative Name:
                DNS:hello-world.hello-world-cluster-namespace.svc.testcluster.local, DNS:hello-world-server-0.hello-world-nodes.hello-world-cluster-namespace.svc.testcluster.local, DNS:hello-world-server-1.hello-world-nodes.hello-world-cluster-namespace.svc.testcluster.local
    Signature Algorithm: sha256WithRSAEncryption
         53:de:00:c7:61:8d:74:b6:fc:9c:9e:1e:f0:39:8e:93:09:20:
         ...

Step 3: Do NOT Obtain cerificate with openssl !!!

typical command for self-signed certificate

openssl x509 -req -days 365 -in server-key.csr -signkey server-key.key -out server-key.crt

or typical command for certificate signed with external key

openssl x509 -req -days 365 -in server-key.csr -signkey /etc/kubernetes/pki/ca.key -out server-key.crt

Step 4: Create a Kubernetes Signing Request

follow the steps of Create Signing Request paragraph

Step 5: Create a Kubernetes secret

follow the steps of Create a Kubernetes secret paragraph
- do not forget -n hello-world-cluster-namespace

kubectl create secret tls hello-world-tls-secret -n hello-world-cluster-namespace  --cert=server.crt --key=server-key.pem

Step 6: Recreate RabbitMQ cluster

follow the steps of Recreate RabbitMQ cluster paragraph

U1.50 Ubuntu Quick Start (QS): RabbitMq Kubernetes cluster operator. - chempkovsky/CS2WPF-and-CS2XAMARIN GitHub Wiki

Reading

We start with

Install RabbitMq cluster operator

RabbitMq cluster operator secrets

Uninstall RabbitMq cluster operator

Insufficient memory for Simple RabbitMq cluster

Create Simple RabbitMq cluster

Status of the created RabbitMq cluster

Troubleshooting

Pod memory consumption

Default storage class

Replicas

Requests and Limits

RabbitMQ Management UI

Scale the cluster

Allowed namespaces

Get a Shell to a Running Container

Scheduling pods on selected nodes

TLS

Kubernetes Related Articles

RabbitMq Related Articles

Reminder for RabbitMQ on bare metal

Define names

Configure TLS with tls_gen

Install tls_gen

modify openssl.cnf

Generate files

Generated files

Configure TLS with CFSSL

Install CFSSL

Generate csr and pem files

Create Signing Request

Create a Kubernetes secret

Recreate RabbitMQ cluster

Configure TLS with openssl

Step 1: create a private key

Step 2: create Certificate signing request (CSR)

to solve this problem

Step 3: Do NOT Obtain cerificate with openssl !!!

Step 4: Create a Kubernetes Signing Request

Step 5: Create a Kubernetes secret

Step 6: Recreate RabbitMQ cluster

⚠️ **GitHub.com Fallback** ⚠️

⚠️ GitHub.com Fallback ⚠️