2) Redis Operator for Redis Cluster (by opstree) - jangjaelee/tutorials-redis-operator GitHub Wiki

2022.06. 이장재 📧 [email protected] 📂 https://github.com/jangjaelee 📒 http://www.awx.kr

Introduction

Opstree Solutions에서 제작한 Redis Operator는 Kubernetes cluster에서 Redis Standalone/Cluster mode 설정을 만들고 감독하는 Open Source를 지원하는 Redis Community를 위한 Golang-based Redis Operator 입니다.

Cloud 및 Bare-metal 환경에서 모범 사례를 사용하여 Redis Cluster 설정을 생성할 수 있으며, redis-exporter를 사용하여 내장된 모니터링 기능을 제공합니다.

지원하는 Redis 설정 유형은 다음과 같습니다.

Redis Standalone
Redis Cluster (in-build leader follower with sharding and replication mode)

Supported Features

Redis Operator가 지원하는 기능으로 다음과 같은 것들이 있습니다.

Redis Cluster (sharding 및 replication mode의 leader-follower) 및 Standalone 설정
Redis Exporter를 사용한 내장 모니터링 지원으로 자세한 통찰력 제공
PVC Template을 사용한 동적 스토리지 프로비저닝
Redis 구성의 성능 조정 모범 사례
Cluster 및 Standalone 설정에 대한 암호 및 암호 없는 지원
다음과 같은 몇 가지 Kubernetes 기반 object 지원
- NodeSelector
- Affinity
- Resource and Limits
- Tolerations
- SecutiryContext
- Storage

Architecture

Prerequisites

Kubernetes cluster v1.11+
kubectl command-line tool
helm command-line tool v3.0+
Persistent Volume
Prometheus Operator (option - to use ServiceMonitor)
Grafana (option - to view metric on Grafana Dashboard)
External Load Balancer (option - to use External Endpoint of Redis) - 필자는 MetalLB를 사용함

✅ Prometheus Operator 설치와 소개는 아래 링크를 참조하세요.

Prometheus Operator for Beginner

✅ MetalLB 설치와 소개는 아래 링크를 참조하세요.

MetalLB

Step 1. Installation Redis Operator

Redis Operator 설치는 YAML manifest와 Helm chart를 이용한 두 가지 설치 방법을 제공하고 있으며 편한 방법을 사용하여 Redis Operator를 설치하세요.

Redis와 redis-exporter의 기본 버전은 다음과 같습니다.

Redis v6.2.5 (default)
redis-exporter v1.0 (default)

Step 1.1. using YAML manifest

제공되는 YAML manifest를 사용하여 Redis Operator를 설치하는 방법 입니다.

manager.yaml에 namespace 생성을 위한 manifest 내용이 포함되어 있어 redis-operator namespace가 함께 생성 됩니다.

$ kubectl apply -f https://raw.githubusercontent.com/OT-CONTAINER-KIT/redis-operator/master/config/crd/bases/redis.redis.opstreelabs.in_redis.yaml
$ kubectl apply -f https://raw.githubusercontent.com/OT-CONTAINER-KIT/redis-operator/master/config/crd/bases/redis.redis.opstreelabs.in_redisclusters.yaml
$ kubectl apply -f https://raw.githubusercontent.com/OT-CONTAINER-KIT/redis-operator/master/config/rbac/serviceaccount.yaml
$ kubectl apply -f https://raw.githubusercontent.com/OT-CONTAINER-KIT/redis-operator/master/config/rbac/role.yaml
$ kubectl apply -f https://raw.githubusercontent.com/OT-CONTAINER-KIT/redis-operator/master/config/rbac/role_binding.yaml
$ kubectl apply -f https://raw.githubusercontent.com/OT-CONTAINER-KIT/redis-operator/master/config/manager/manager.yaml

Step 1.2. using Helm chart

Redis Operator를 좀더 쉽게 하기 위해 Helm chart를 제공하며 values.yaml 파일을 사용하여 사용자 정의할 수 있습니다.

또는 helm show 명령을 사용하여 기본 values를 파일로 생성하여 사용할 수 있습니다.

$ helm show values ot-helm/redis-operator > values.yaml

Redis Operator Helm chart 저장소를 등록하고 설치합니다.

$ helm repo add ot-helm https://ot-container-kit.github.io/helm-charts/
$ helm repo update
$ helm install redis-operator ot-helm/redis-operator -n redis-operator --create-namespace

Step 1.3. See installed objects

이제 operator 설치를 완료 하였으니 생성된 objects과 구동중인 pod를 확인합니다.

$ kubectl get all -n redis-operator
NAME                                  READY   STATUS    RESTARTS      AGE
pod/redis-operator-78c6c95b78-wzkrb   1/1     Running   1             50s

NAME                             READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/redis-operator   1/1     1            1           1m

NAME                                        DESIRED   CURRENT   READY   AGE
replicaset.apps/redis-operator-78c6c95b78   1         1         1       1m

생성된 CRD도 함께 확인하면 redis와 redisclusters CRD들이 생성된 것을 알 수 있습니다.

redis CRD는 Redis Standalone 설정 유형을 위함이며, redisclusters CRD는 Redis Cluster 설정 유형을 위한 것 입니다.

$ kubectl api-resources | egrep '^NAME|redis'
NAME                              SHORTNAMES         APIVERSION                             NAMESPACED   KIND
redis                                                redis.redis.opstreelabs.in/v1beta1     true         Redis
redisclusters                                        redis.redis.opstreelabs.in/v1beta1     true         RedisCluster

kubectl plugin manager인 Krew의 get-all plugins을 설치하면 좀더 쉽게 생성된 objects를 볼 수 있습니다.

🔗 https://github.com/corneliusweig/ketall

$ kubectl get-all -n redis-operator
NAME                                                     NAMESPACE       AGE
configmap/6cab913b.redis.opstreelabs.in                  redis-operator  9s
configmap/kube-root-ca.crt                               redis-operator  18s
pod/redis-operator-78c6c95b78-wzkrb                      redis-operator  18s
secret/default-token-gjvhs                               redis-operator  18s
secret/redis-operator-token-6n5hd                        redis-operator  18s
secret/sh.helm.release.v1.redis-operator.v1              redis-operator  18s
serviceaccount/default                                   redis-operator  18s
serviceaccount/redis-operator                            redis-operator  18s
deployment.apps/redis-operator                           redis-operator  18s
replicaset.apps/redis-operator-78c6c95b78                redis-operator  18s
lease.coordination.k8s.io/6cab913b.redis.opstreelabs.in  redis-operator  9s

Step 2. Deploy Redis and Configuration

이번 단계에서는 Redis를 배포하고 CR의 구성을 한번 살펴 보겠습니다.

설치에 앞서 Redis 내에서 password 기반 인증을 사용하려는 경우 secret을 사용하여 사용할 수 있으며, secret의 기본 이름은 redis-secret 이며, key 이름은 password 입니다. 그리고 helm chart를 사용한다면 override도 가능합니다.

redis auth를 위한 secret object를 생성 합니다.

$ kubectl create secret generic redis-secret --from-literal=password=redispassword -n redis-system
secret/redis-secret created

$ kubectl describe secret/redis-secret -n redis-system
Name:         redis-secret
Namespace:    redis-system
Labels:       <none>
Annotations:  <none>

Type:  Opaque

Data
====
password:  13 bytes

Step 2.1. Redis Standalone

Redis Standalone mode에서는 Redis를 단일 Pod로 배포하므로 StatefulSets의 replicas: 1 입니다. 그리고 설정이 간편하고 복잡성이 없지만, 고가용성(HA) 및 복원력(resilience)은 제공되지 않습니다.

다음 명령을 사용하여 Redis를 Standalone mode로 배포 합니다.

Prometheus Operator의 ServiceMonitor CR을 생성하기 위해 --set serviceMonitor.enabled=true value를 사용하였으며 Prometheus Operator를 사용하지 않는다면 해당 value는 생략 가능합니다.

🔗 Redis Standalone의 Helm chart 내용은 [링크]에서 확인 가능합니다.

$ helm install redis-standalone ot-helm/redis \
-n redis-system --create-system \
--set redisCluster.redisSecret.secretName="redis-secret" \
--set redisCluster.redisSecret.secretKey="password" \
--set serviceMonitor.enabled=true \
--set serviceMonitor.namespace="redis-system"

배포된 objects를 확인해보겠습니다.

$ kubectl get-all -n redis-system
NAME                                                                         NAMESPACE     AGE
configmap/kube-root-ca.crt                                                   redis-system  1m
endpoints/redis-standalone                                                   redis-system  1m
endpoints/redis-standalone-headless                                          redis-system  1m
persistentvolumeclaim/redis-standalone-redis-standalone-0                    redis-system  1m
pod/redis-standalone-0                                                       redis-system  1m
secret/default-token-kbmwd                                                   redis-system  1m
secret/redis-secret                                                          redis-system  1m
secret/sh.helm.release.v1.redis-standalone.v1                                redis-system  1m
secret/sh.helm.release.v1.redis-standalone.v2                                redis-system  1m
serviceaccount/default                                                       redis-system  1m
service/redis-standalone                                                     redis-system  1m
service/redis-standalone-headless                                            redis-system  1m
controllerrevision.apps/redis-standalone-7fbb5dd9fc                          redis-system  1m
statefulset.apps/redis-standalone                                            redis-system  1m
endpointslice.discovery.k8s.io/redis-standalone-hccd9                        redis-system  1m
endpointslice.discovery.k8s.io/redis-standalone-headless-rvxhs               redis-system  1m
servicemonitor.monitoring.coreos.com/redis-standalone-prometheus-monitoring  redis-system  1m
redis.redis.redis.opstreelabs.in/redis-standalone                            redis-system  1m

Redis Operator Controller는 redis CR의 생성을 감지하여 StatefulSet object를 자동으로 생성하게 됩니다.

Redis Standalone을 구성하는 redis CR의 configuration은 다음의 링크에서 Helm Parameters를 사용하여 설정이 가능합니다.

🔗 https://ot-container-kit.github.io/redis-operator/guide/redis-config.html

Step 2.2. Redis Cluster

Redis Cluster는 데이터 샤딩 전략으로 여러 Redis node에 데이터를 자동으로 분할해줍니다. 분산 스토리지를 구현하고 단일 장애 지점(SPOF) 방지하는 Redis의 고급 기능 입니다.

다음 명령을 사용하여 Redis를 Cluster mode로 배포 합니다.

🔗 Redis Cluster의 Helm chart 내용은 [링크]에서 확인 가능합니다.

$ helm install redis-cluster ot-helm/redis-cluster \
-n redis-system --create-system \
--set redisCluster.clusterSize=3 \
--set redisCluster.leader.replicas=3 \
--set redisCluster.follower.replicas=3 \
--set redisCluster.redisSecret.secretName="redis-secret" \
--set redisCluster.redisSecret.secretKey="password" \
--set serviceMonitor.enabled=true \
--set serviceMonitor.namespace="redis-system"

이제 배포된 objects를 확인해보겠습니다.

$ kubectl get all -n redis-system
NAME                           READY   STATUS    RESTARTS   AGE
pod/redis-cluster-follower-0   2/2     Running   0          4m4s
pod/redis-cluster-follower-1   2/2     Running   0          3m46s
pod/redis-cluster-follower-2   2/2     Running   0          3m13s
pod/redis-cluster-leader-0     2/2     Running   0          3m10s
pod/redis-cluster-leader-1     2/2     Running   0          2m46s
pod/redis-cluster-leader-2     2/2     Running   0          2m11s

NAME                                      TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
service/redis-cluster-follower            ClusterIP   10.103.232.239   <none>        6379/TCP,9121/TCP   4m4s
service/redis-cluster-follower-headless   ClusterIP   None             <none>        6379/TCP            4m4s
service/redis-cluster-leader              ClusterIP   10.101.133.245   <none>        6379/TCP,9121/TCP   3m10s
service/redis-cluster-leader-headless     ClusterIP   None             <none>        6379/TCP            3m10s

NAME                                      READY   AGE
statefulset.apps/redis-cluster-follower   3/3     4m
statefulset.apps/redis-cluster-leader     3/3     4m

$ kubectl get rediscluster -n redis-system
NAME            CLUSTERSIZE   LEADERREPLICAS   FOLLOWERREPLICAS   AGE
redis-cluster   3             3                3                  1m

$ kubectl get-all -n redis-system
NAME                                                                               NAMESPACE     AGE
configmap/kube-root-ca.crt                                                         redis-system  14h
endpoints/redis-cluster-follower                                                   redis-system  88m
endpoints/redis-cluster-follower-headless                                          redis-system  88m
endpoints/redis-cluster-leader                                                     redis-system  88m
endpoints/redis-cluster-leader-headless                                            redis-system  88m
persistentvolumeclaim/redis-cluster-follower-redis-cluster-follower-0              redis-system  88m
persistentvolumeclaim/redis-cluster-follower-redis-cluster-follower-1              redis-system  88m
persistentvolumeclaim/redis-cluster-follower-redis-cluster-follower-2              redis-system  88m
persistentvolumeclaim/redis-cluster-leader-redis-cluster-leader-0                  redis-system  88m
persistentvolumeclaim/redis-cluster-leader-redis-cluster-leader-1                  redis-system  88m
persistentvolumeclaim/redis-cluster-leader-redis-cluster-leader-2                  redis-system  88m
pod/redis-cluster-follower-0                                                       redis-system  88m
pod/redis-cluster-follower-1                                                       redis-system  88m
pod/redis-cluster-follower-2                                                       redis-system  87m
pod/redis-cluster-leader-0                                                         redis-system  88m
pod/redis-cluster-leader-1                                                         redis-system  87m
pod/redis-cluster-leader-2                                                         redis-system  87m
secret/default-token-kbmwd                                                         redis-system  14h
secret/redis-secret                                                                redis-system  14h
secret/sh.helm.release.v1.redis-cluster.v1                                         redis-system  91m
serviceaccount/default                                                             redis-system  14h
service/redis-cluster-follower                                                     redis-system  88m
service/redis-cluster-follower-headless                                            redis-system  88m
service/redis-cluster-leader                                                       redis-system  88m
service/redis-cluster-leader-headless                                              redis-system  88m
controllerrevision.apps/redis-cluster-follower-5444b5bdcf                          redis-system  88m
controllerrevision.apps/redis-cluster-leader-5699997689                            redis-system  88m
statefulset.apps/redis-cluster-follower                                            redis-system  88m
statefulset.apps/redis-cluster-leader                                              redis-system  88m
endpointslice.discovery.k8s.io/redis-cluster-follower-b4vsl                        redis-system  88m
endpointslice.discovery.k8s.io/redis-cluster-follower-headless-l46zg               redis-system  88m
endpointslice.discovery.k8s.io/redis-cluster-leader-headless-dtskw                 redis-system  88m
endpointslice.discovery.k8s.io/redis-cluster-leader-xb2cd                          redis-system  88m
servicemonitor.monitoring.coreos.com/redis-cluster-follower-prometheus-monitoring  redis-system  88m
servicemonitor.monitoring.coreos.com/redis-cluster-leader-prometheus-monitoring    redis-system  88m
rediscluster.redis.redis.opstreelabs.in/redis-cluster                              redis-system  88m

Redis Standalone와는 다르게 Redis Cluster mode에서는 rediscluster CR이 생성되면 Leader와 Follow를 위한 StatefulSet이 각각 배포 된것을 알 수 있으며, 그에 따라 Service와 Prometheus Operator의 ServiceMonitor도 함께 생성 되었습니다.

Redis leader pod에 연결하여 redis cluster 정보와 node 연결 상태를 확인합니다.

$ kubectl exec -it redis-cluster-leader-0 -n redis-system -- redis-cli -a redispassword cluster info
Defaulted container "redis-cluster-leader" out of: redis-cluster-leader, redis-exporter
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:3
cluster_my_epoch:1
cluster_stats_messages_ping_sent:376
cluster_stats_messages_pong_sent:375
cluster_stats_messages_sent:751
cluster_stats_messages_ping_received:372
cluster_stats_messages_pong_received:376
cluster_stats_messages_meet_received:3
cluster_stats_messages_received:751

$ kubectl exec -it redis-cluster-leader-0 -n redis-system -- redis-cli -a redispassword cluster nodes
Defaulted container "redis-cluster-leader" out of: redis-cluster-leader, redis-exporter
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
5483f9be46324b9e8f48b431fc4128c788525cdc 172.16.10.103:6379@16379 master - 0 1655908354065 2 connected 5461-10922
4d0bd6e99c00cddd6f9858da371526f39fef157a 172.16.10.75:6379@16379 slave 5483f9be46324b9e8f48b431fc4128c788525cdc 0 1655908352043 2 connected
87fe6e67e43ddb9f9ad6acd830d360b83250c630 172.16.84.121:6379@16379 slave 7ded434ed7be8db05a7851dc44e082e38bbf398b 0 1655908353559 1 connected
a956b4aaf4a04957bc6b0a42bdb62122c91fb44e 172.16.84.115:6379@16379 slave 742149aba5d2e572b6f85ea59a964880fa79b367 0 1655908352545 3 connected
7ded434ed7be8db05a7851dc44e082e38bbf398b 172.16.84.122:6379@16379 myself,master - 0 1655908351000 1 connected 0-5460
742149aba5d2e572b6f85ea59a964880fa79b367 172.16.78.67:6379@16379 master - 0 1655908352547 3 connected 10923-16383

redis cluster nodes 상태에서 IP 정보와 Redis instance의 pod IP를 대조하면 어떤 leader와 follower가 각각 어떤 pod끼리 master-replica로 연결되어 있는지 알 수 있습니다.

$ kubectl get pod -n redis-system -o wide
NAME                       READY   STATUS    RESTARTS   AGE     IP              NODE            NOMINATED NODE   READINESS GATES
redis-cluster-follower-0   2/2     Running   0          6m1s    172.16.84.121   k8s-w3-dragon   <none>           <none>
redis-cluster-follower-1   2/2     Running   0          7m38s   172.16.10.75    k8s-w2-dragon   <none>           <none>
redis-cluster-follower-2   2/2     Running   0          7m5s    172.16.84.115   k8s-w3-dragon   <none>           <none>
redis-cluster-leader-0     2/2     Running   0          6m16s   172.16.84.122   k8s-w3-dragon   <none>           <none>
redis-cluster-leader-1     2/2     Running   0          7m53s   172.16.10.103   k8s-w2-dragon   <none>           <none>
redis-cluster-leader-2     2/2     Running   0          7m20s   172.16.78.67    k8s-w1-dragon   <none>           <none>

그리고 pod의 log를 찍어보면 master-replica의 replication 상태를 확인할 수 있습니다.

# Redis leader-0 node (master-0) 

$ kubectl logs -f pod/redis-cluster-leader-0 -n redis-system -c redis-cluster-leader
Starting redis service in cluster mode.....
11:C 22 Jun 2022 14:27:20.230 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
11:C 22 Jun 2022 14:27:20.230 # Redis version=6.2.5, bits=64, commit=00000000, modified=0, pid=11, just started
11:C 22 Jun 2022 14:27:20.230 # Configuration loaded
11:M 22 Jun 2022 14:27:20.230 * monotonic clock: POSIX clock_gettime
11:M 22 Jun 2022 14:27:20.231 * Node configuration loaded, I'm 7ded434ed7be8db05a7851dc44e082e38bbf398b
11:M 22 Jun 2022 14:27:20.231 * Running mode=cluster, port=6379.
11:M 22 Jun 2022 14:27:20.231 # Server initialized
11:M 22 Jun 2022 14:27:20.231 * Ready to accept connections
11:M 22 Jun 2022 14:28:40.238 # configEpoch set to 1 via CLUSTER SET-CONFIG-EPOCH
11:M 22 Jun 2022 14:28:45.207 # Cluster state changed: ok
11:M 22 Jun 2022 14:28:57.589 * Replica 172.16.84.121:6379 asks for synchronization
11:M 22 Jun 2022 14:28:57.589 * Partial resynchronization not accepted: Replication ID mismatch (Replica asked for '8e3ffd79b9f7370b05effda9a837c2956e5fabed', my replication IDs are 'd8bea88fcebd4c90442116e74a15a7fd2a3935d4' and '0000000000000000000000000000000000000000')
11:M 22 Jun 2022 14:28:57.589 * Replication backlog created, my new replication IDs are '8e2dab8037c5e9c194cf0c659c9c28f77c08614e' and '0000000000000000000000000000000000000000'
11:M 22 Jun 2022 14:28:57.589 * Starting BGSAVE for SYNC with target: disk
11:M 22 Jun 2022 14:28:57.589 * Background saving started by pid 109
109:C 22 Jun 2022 14:28:57.591 * DB saved on disk
109:C 22 Jun 2022 14:28:57.591 * RDB: 0 MB of memory used by copy-on-write
11:M 22 Jun 2022 14:28:57.619 * Background saving terminated with success
11:M 22 Jun 2022 14:28:57.619 * Synchronization with replica 172.16.84.121:6379 succeeded

# Redis follower-0 node (replica-0)

$ kubectl logs -f pod/redis-cluster-follower-0 -n redis-system -c redis-cluster-follower
Starting redis service in cluster mode.....
13:C 22 Jun 2022 14:27:35.559 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
13:C 22 Jun 2022 14:27:35.559 # Redis version=6.2.5, bits=64, commit=00000000, modified=0, pid=13, just started
13:C 22 Jun 2022 14:27:35.559 # Configuration loaded
13:M 22 Jun 2022 14:27:35.560 * monotonic clock: POSIX clock_gettime
13:M 22 Jun 2022 14:27:35.560 * Node configuration loaded, I'm 87fe6e67e43ddb9f9ad6acd830d360b83250c630
13:M 22 Jun 2022 14:27:35.561 * Running mode=cluster, port=6379.
13:M 22 Jun 2022 14:27:35.561 # Server initialized
13:M 22 Jun 2022 14:27:35.561 * Ready to accept connections
13:S 22 Jun 2022 14:28:57.588 * Before turning into a replica, using my own master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer.
13:S 22 Jun 2022 14:28:57.588 * Connecting to MASTER 172.16.84.122:6379
13:S 22 Jun 2022 14:28:57.588 * MASTER <-> REPLICA sync started
13:S 22 Jun 2022 14:28:57.588 # Cluster state changed: ok
13:S 22 Jun 2022 14:28:57.588 * Non blocking connect for SYNC fired the event.
13:S 22 Jun 2022 14:28:57.589 * Master replied to PING, replication can continue...
13:S 22 Jun 2022 14:28:57.589 * Trying a partial resynchronization (request 8e3ffd79b9f7370b05effda9a837c2956e5fabed:1).
13:S 22 Jun 2022 14:28:57.590 * Full resync from master: 8e2dab8037c5e9c194cf0c659c9c28f77c08614e:0
13:S 22 Jun 2022 14:28:57.590 * Discarding previously cached master state.
13:S 22 Jun 2022 14:28:57.619 * MASTER <-> REPLICA sync: receiving 175 bytes from master to disk
13:S 22 Jun 2022 14:28:57.619 * MASTER <-> REPLICA sync: Flushing old data
13:S 22 Jun 2022 14:28:57.619 * MASTER <-> REPLICA sync: Loading DB in memory
13:S 22 Jun 2022 14:28:57.620 * Loading RDB produced by version 6.2.5
13:S 22 Jun 2022 14:28:57.620 * RDB age 0 seconds
13:S 22 Jun 2022 14:28:57.620 * RDB memory usage when created 2.60 Mb
13:S 22 Jun 2022 14:28:57.620 * MASTER <-> REPLICA sync: Finished with success
13:S 22 Jun 2022 14:28:57.621 * Background append only file rewriting started by pid 83
13:S 22 Jun 2022 14:28:57.650 * AOF rewrite child asks to stop sending diffs.
83:C 22 Jun 2022 14:28:57.650 * Parent agreed to stop sending diffs. Finalizing AOF...
83:C 22 Jun 2022 14:28:57.650 * Concatenating 0.00 MB of AOF diff received from parent.
83:C 22 Jun 2022 14:28:57.651 * SYNC append only file rewrite performed
83:C 22 Jun 2022 14:28:57.651 * AOF rewrite: 0 MB of memory used by copy-on-write
13:S 22 Jun 2022 14:28:57.725 * Background AOF rewrite terminated with success
13:S 22 Jun 2022 14:28:57.725 * Residual parent diff successfully flushed to the rewritten AOF (0.00 MB)
13:S 22 Jun 2022 14:28:57.725 * Background AOF rewrite finished successfully

Redis Cluster을 구성하는 rediscluster CR의 configuration은 다음의 링크에서 Helm Parameters를 사용하여 설정이 가능합니다.

🔗 https://ot-container-kit.github.io/redis-operator/guide/redis-cluster-config.html

Step 2.3. Exposing Service (External Endpoint)

Redis Operator는 기본적으로 Redis Standalone와 Cluster 모두 Service Endpoint는 Kubernetes cluster내로 제한됩니다. 그러나 Service object의 Type을 NodePort 또는 LoadBalancer 유형으로 서비스를 외부노출 시킬 수 있습니다.

Redis Standalone와 Cluster는 Helm chart에 external Service를 위한 구성 설정을 helm value를 통해 제공하고 있어 Redis를 Kubernetes cluster 외부로 노출시킬 수 있습니다.

externalService:
  enabled: true
  annotations:
    metallb.universe.tf/address-pool: "default"
  serviceType: LoadBalancer
  port: 6379

또는, helm 명령어의 set value를 사용하여 CLI에서도 구성이 가능합니다.

Redis Cluster의 Redis를 외부노출 시켜 보겠습니다.

$ helm upgrade redis-cluster ot-helm/redis-cluster \
-n redis-system --install \
--set redisCluster.clusterSize=3 \
--set redisCluster.leader.replicas=3 \
--set redisCluster.follower.replicas=3 \
--set redisCluster.redisSecret.secretName="redis-secret" \
--set redisCluster.redisSecret.secretKey="password" \
--set serviceMonitor.enabled=true \
--set serviceMonitor.namespace="redis-system" \
--set externalService.enabled=true \
--set externalService.annotation."metallb\.universe\.tf/address-pool"="default"

externalService.annotation에는 external LoadBalancer를 명시하는 부분으로 사용하는 LoadBalancer를 확인후 적절하게 기재하여 사용하시기 바랍니다. 필자는 MetalLB를 사용 하고 있습니다.

service object 이름은 StatefulSets 이름을 prefix로 사용하여 <statefulset name>-external-service 와 같이 생성 됩니다.

$ kubectl get svc -n redis-system | egrep '^NAME|external-service'
NAME                                      TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)             AGE
redis-cluster-follower-external-service   LoadBalancer   10.105.17.6     192.168.1.184   6379:32311/TCP      19m
redis-cluster-leader-external-service     LoadBalancer   10.96.173.154   192.168.1.185   6379:32114/TCP      19m

external Service를 생성하고 redis-cli로 서버에 접속하면 접속이 안됩니다.

$ redis-cli -h 192.168.1.184 -a redispassword cluster nodes
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.

Could not connect to Redis at 192.168.1.184:6379: No route to host

$ redis-cli -h 192.168.1.185 -a redispassword cluster nodes
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.

Could not connect to Redis at 192.168.1.185:6379: No route to host

이유를 찾아보니 service object에 명시된 Label selector에서 문제를 발견 했습니다.

확인을 해보면 redis-cluster-leader-external-service와 redis-cluster-follow-external-service의 Label selector에 redis_setup_type의 값이 각각 leader와 follower로 되어 있습니다.

$ kubectl get svc/redis-cluster-leader-external-service -n redis-system -o=jsonpath='{.spec.selector}' | jq
{
  "app": "redis-cluster-leader",
  "redis_setup_type": "leader",
  "role": "leader"
}

$ kubectl get svc/redis-cluster-follower-external-service -n redis-system -o=jsonpath='{.spec.selector}' | jq
{
  "app": "redis-cluster-follower",
  "redis_setup_type": "follower",
  "role": "follower"
}

이는 Redis Cluster의 Helm template에서 오류의 원인을 찾을 수 있었습니다. leader와 follow의 external-service를 생성하는 template를 보면 이미 위에서 확인 한바와 같이 redis_setup_type의 값이 leader와 follower로 정의 되어 있습니다.

Helm template - leader-service.yaml
Helm template - follower-service.yaml

그렇지만 Redis Operator Controller가 rediscluster CR의 생성을 감지하고 StatefulSet을 생성 할 때 Label이 자동으로 삽입되는데 Redis Cluster 부분의 controller 소스코드를 보면 redis_setup_type의 label 값이 cluster로 prefix 되어 있었습니다.

결국 Helm chart에 redis_setup_type의 label에 오류가 있다는 것을 확인했고 service object의 Label selector를 변경하는 것으로 해결을 할 수 있었습니다.

$ kubectl patch svc/redis-cluster-leader-external-service -n redis-system -p '{"spec":{"selector":{"redis_setup_type":"cluster"}}}'
service/redis-cluster-leader-external-service patched

$ kubectl patch svc/redis-cluster-follower-external-service -n redis-system -p '{"spec":{"selector":{"redis_setup_type":"cluster"}}}'
service/redis-cluster-follower-external-service patched

$ redis-cli -h 192.168.1.184 -a redispassword cluster nodes
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
ba736d6a0fe48cf54290fd78ac3670079df7fe95 172.16.84.92:6379@16379 slave cb557203cc8779da78234cda345863db6ddca3ba 0 1655878128074 1 connected
7d5511d537ec147a6cb15cdd0c14a8d9d5d73044 172.16.10.88:6379@16379 master - 0 1655878130103 2 connected 5461-10922
162ed69a7cf46f58ff0d6c0dc5e80173a0999c36 172.16.78.111:6379@16379 myself,slave ae65d3acf8c9003ce07496b6d7d5ef4ef9e9f3a9 0 1655878129000 3 connected
39d7391b7ba0085513a5b7ed1d00f66f2cdbd425 172.16.10.93:6379@16379 slave 7d5511d537ec147a6cb15cdd0c14a8d9d5d73044 0 1655878129000 2 connected
cb557203cc8779da78234cda345863db6ddca3ba 172.16.84.81:6379@16379 master - 0 1655878129085 1 connected 0-5460
ae65d3acf8c9003ce07496b6d7d5ef4ef9e9f3a9 172.16.78.84:6379@16379 master - 0 1655878129084 3 connected 10923-16383

$ redis-cli -h 192.168.1.185 -a redispassword cluster nodes
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
7d5511d537ec147a6cb15cdd0c14a8d9d5d73044 172.16.10.88:6379@16379 master - 0 1655878134035 2 connected 5461-10922
cb557203cc8779da78234cda345863db6ddca3ba 172.16.84.81:6379@16379 myself,master - 0 1655878134000 1 connected 0-5460
ae65d3acf8c9003ce07496b6d7d5ef4ef9e9f3a9 172.16.78.84:6379@16379 master - 0 1655878135048 3 connected 10923-16383
162ed69a7cf46f58ff0d6c0dc5e80173a0999c36 172.16.78.111:6379@16379 slave ae65d3acf8c9003ce07496b6d7d5ef4ef9e9f3a9 0 1655878134542 3 connected
39d7391b7ba0085513a5b7ed1d00f66f2cdbd425 172.16.10.93:6379@16379 slave 7d5511d537ec147a6cb15cdd0c14a8d9d5d73044 0 1655878134000 2 connected
ba736d6a0fe48cf54290fd78ac3670079df7fe95 172.16.84.92:6379@16379 slave cb557203cc8779da78234cda345863db6ddca3ba 0 1655878135556 1 connected

✅ Redis Standalone mode는 문제 없이 정상동작 하는 것을 확인했습니다.

Step 3. Monitoring

Step 3.1. Prometheus Monitoring

Redis Operator는 redis-exporter를 사용하여 Prometheus 형식으로 redis 설정의 metric을 노출시킬 수 있으며, Redis Standalone/Cluster 모두 지원합니다.

Helm chart를 통해 redis-exporter를 활성화하여 배포할 수 있으며 사용자 지정 values.yaml 파일을 생성하여 상세한 설정을 override로 구성할 수 있습니다. (redisExporter 활성화 기본값은 true 입니다.)

redisExporter:
  enabled: true
  image: quay.io/opstree/redis-exporter:1.0
  imagePullPolicy: Always
  resources:
    requests:
      cpu: 100m
      memory: 128Mi
    limits:
      cpu: 100m
      memory: 128Mi
  env:
  - name: REDIS_EXPORTER_INCL_SYSTEM_METRICS
    value: "true"
  - name: REDIS_EXPORTER_PING_ON_CONNECT
    value: "true"
  - name: REDIS_EXPORTER_INCL_CONFIG_METRICS
    value: "true"
  - name: REDIS_EXPORTER_EXPORT_CLIENT_LIST
    value: "true"

redis-export를 활성화하고 leader와 follower의 Redis instance pod를 describe해보면 redis-exporter가 sidecar container로 동작하는 것이 보입니다.

$ kubectl describe pod/redis-cluster-leader-0 -n redis-system
Name:         redis-cluster-leader-0
Namespace:    redis-system
Priority:     0
Node:         k8s-w3-dragon/192.168.1.113
Start Time:   Wed, 22 Jun 2022 17:31:09 +0900
.
.
Containers:
redis-exporter:
    Container ID:   containerd://7acdf990af25a47975938d3a528c0c80c08502558f55c3b9e8a25df9922b10a0
    Image:          quay.io/opstree/redis-exporter:1.0
    Image ID:       quay.io/opstree/redis-exporter@sha256:d52aebf0ff162b5b953dfe7dfcb8a319748654fd7325007d93d2884e027edd2d
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Wed, 22 Jun 2022 17:31:11 +0900
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     100m
      memory:  128Mi
    Requests:
      cpu:     100m
      memory:  128Mi
    Environment:
      PERSISTENCE_ENABLED:                 true
      REDIS_ADDR:                          redis://localhost:6379
      REDIS_EXPORTER_EXPORT_CLIENT_LIST:   true
      REDIS_EXPORTER_INCL_CONFIG_METRICS:  true
      REDIS_EXPORTER_INCL_SYSTEM_METRICS:  true
      REDIS_EXPORTER_INCL_SYSTEM_METRICS:  true
      REDIS_EXPORTER_PING_ON_CONNECT:      true
      REDIS_PASSWORD:                      <set to the key 'password' in secret 'redis-secret'>  Optional: false
      SERVER_MODE:                         cluster
      SETUP_MODE:                          cluster
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-kdwfs (ro)
.
.
.
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  45m   default-scheduler  Successfully assigned redis-system/redis-cluster-leader-0 to k8s-w3-dragon
  Normal  Pulled     45m   kubelet            Container image "quay.io/opstree/redis:v6.2.5" already present on machine
  Normal  Created    45m   kubelet            Created container redis-cluster-leader
  Normal  Started    45m   kubelet            Started container redis-cluster-leader
  Normal  Pulling    45m   kubelet            Pulling image "quay.io/opstree/redis-exporter:1.0"
  Normal  Pulled     44m   kubelet            Successfully pulled image "quay.io/opstree/redis-exporter:1.0" in 841.273976ms
  Normal  Created    44m   kubelet            Created container redis-exporter
  Normal  Started    44m   kubelet            Started container redis-exporter

Step 3.2. ServiceMonitor(of Prometheus Operator)

redis-exporter 배포 및 구성이 완료 되면 endpoint를 모니터링하기 위해 Prometheus Target Configuration 업데이트 해야 합니다.

service endpoint를 수동으로 Prometheus 대상 구성하는 것은 보통 어려운게 아닙니다. 그래서 Prometheus Operator의 ServiceMonitor CR를 사용하여 endpoint들에 대한 대상 구성을 자동으로 설정 할 수 있으며, Redis Standalone/Cluster의 Helm chart에서 ServiceMonitor를 지원하고 있어 사용자 지정 value.yaml 파일을 생성하여 override 방식으로 구성할 수 있습니다.

serviceMonitor:
  enabled: false
  interval: 30s
  scrapeTimeout: 10s
  namespace: redis-system # redis instance와 service object가 배포된 namespace

또는, ServiceMonitor CR을 yaml 파일로 정의하여 적용할 수 있습니다.

Redis Standalone

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    app.kubernetes.io/component: middleware
    app.kubernetes.io/instance: redis-cluster
    app.kubernetes.io/name: redis-cluster
    app.kubernetes.io/version: 0.10.0
  name: redis-standalone-prometheus-monitoring
  namespace: redis-system
spec:
  endpoints:
  - interval: 30s
    port: redis-exporter
    scrapeTimeout: 10s
  namespaceSelector:
    matchNames:
    - redis-system
  selector:
    matchLabels:
      app: redis-standalone
      redis_setup_type: standalone

Redis Cluster

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    app.kubernetes.io/component: middleware
    app.kubernetes.io/instance: redis-cluster
    app.kubernetes.io/name: redis-cluster
    app.kubernetes.io/version: 0.10.0
  name: redis-cluster-leader-prometheus-monitoring
  namespace: redis-system
spec:
  endpoints:
  - interval: 30s
    port: redis-exporter
    scrapeTimeout: 10s
  namespaceSelector:
    matchNames:
    - redis-system
  selector:
    matchLabels:
      app: redis-cluster-leader
      redis_setup_type: cluster
      role: leader

Prometheus Dashboard에 접속하면 Targets이 추가되었고 지표(metric)들을 수집되고 있습니다.

Step 3.3. Grafana Dashboard

Redis Cluster의 통계를 모니터링하기 위해 Grafana Dashboard를 제공하고 있어 아래 링크에서 다운로드 받아 Grafana에 등록 하여 사용하시면 됩니다.

🔗 https://ot-container-kit.github.io/redis-operator/guide/grafana.html

Step 4. Redis Cluster Failover Testing

Redis Cluster에 master node 3개, replica node 3개가 있는 상황에서 1번 master node (redis-cluster-leader-0)를 삭제하면, 1번 replica node (redis-cluster-follower-01)가 master로 승격되어 지속적으로 데이터의 일관성을 보장합니다. 하지만 cluster-node-time 시간을 포함해서 정상으로 돌아오기까지는 몇 초의 시간이 소요됩니다.

redis-cli 명령을 사용하여 Redis Cluster의 1번 master node에서 dummy data를 write 합니다.

$ kubectl exec -it redis-cluster-leader-0 -n redis-system -c redis-cluster-leader -- redis-cli -a redispassword -c set doik gradute
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
OK

그리고 다른 master(leader)와 replica(follower) nodes에서 data를 read 하면 동일한 데이터를 읽을 수 있습니다.

$ kubectl exec -it redis-cluster-follower-0 -n redis-system -c redis-cluster-follower -- redis-cli -a redispassword -c get doik
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
"gradute"

$ kubectl exec -it redis-cluster-leader-2 -n redis-system -c redis-cluster-leader -- redis-cli -a redispassword -c get doik
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
"gradute"

$ kubectl exec -it redis-cluster-follower-1 -n redis-system -c redis-cluster-follower -- redis-cli -a redispassword -c get doik
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
"gradute"

dummy data를 write후에 redis log를 보면 master node에서 data writeing transition이 발생과 동시에 replica node에 data가 replication 되었다는 것을 알 수 있습니다.

master 1번 노드(redis-cluster-leader-0 pod)의 redis log

$ kubectl logs -f redis-cluster-leader-0 -n redis-system
Defaulted container "redis-cluster-leader" out of: redis-cluster-leader, redis-exporter
Starting redis service in cluster mode.....
11:C 22 Jun 2022 14:27:20.230 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
11:C 22 Jun 2022 14:27:20.230 # Redis version=6.2.5, bits=64, commit=00000000, modified=0, pid=11, just started
11:C 22 Jun 2022 14:27:20.230 # Configuration loaded
11:M 22 Jun 2022 14:27:20.230 * monotonic clock: POSIX clock_gettime
11:M 22 Jun 2022 14:27:20.231 * Node configuration loaded, I'm 7ded434ed7be8db05a7851dc44e082e38bbf398b
11:M 22 Jun 2022 14:27:20.231 * Running mode=cluster, port=6379.
11:M 22 Jun 2022 14:27:20.231 # Server initialized
11:M 22 Jun 2022 14:27:20.231 * Ready to accept connections
11:M 22 Jun 2022 14:28:40.238 # configEpoch set to 1 via CLUSTER SET-CONFIG-EPOCH
11:M 22 Jun 2022 14:28:45.207 # Cluster state changed: ok
11:M 22 Jun 2022 14:28:57.589 * Replica 172.16.84.121:6379 asks for synchronization
11:M 22 Jun 2022 14:28:57.589 * Partial resynchronization not accepted: Replication ID mismatch (Replica asked for '8e3ffd79b9f7370b05effda9a837c2956e5fabed', my replication IDs are 'd8bea88fcebd4c90442116e74a15a7fd2a3935d4' and '0000000000000000000000000000000000000000')
11:M 22 Jun 2022 14:28:57.589 * Replication backlog created, my new replication IDs are '8e2dab8037c5e9c194cf0c659c9c28f77c08614e' and '0000000000000000000000000000000000000000'
11:M 22 Jun 2022 14:28:57.589 * Starting BGSAVE for SYNC with target: disk
11:M 22 Jun 2022 14:28:57.589 * Background saving started by pid 109
109:C 22 Jun 2022 14:28:57.591 * DB saved on disk
109:C 22 Jun 2022 14:28:57.591 * RDB: 0 MB of memory used by copy-on-write
11:M 22 Jun 2022 14:28:57.619 * Background saving terminated with success
11:M 22 Jun 2022 14:28:57.619 * Synchronization with replica 172.16.84.121:6379 succeeded

replica 1번 노드(redis-cluster-follower-0 pod)의 redis log

$ kubectl logs -f redis-cluster-follower-0 -n redis-system
Defaulted container "redis-cluster-follower" out of: redis-cluster-follower, redis-exporter
Starting redis service in cluster mode.....
13:C 22 Jun 2022 14:27:35.559 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
13:C 22 Jun 2022 14:27:35.559 # Redis version=6.2.5, bits=64, commit=00000000, modified=0, pid=13, just started
13:C 22 Jun 2022 14:27:35.559 # Configuration loaded
13:M 22 Jun 2022 14:27:35.560 * monotonic clock: POSIX clock_gettime
13:M 22 Jun 2022 14:27:35.560 * Node configuration loaded, I'm 87fe6e67e43ddb9f9ad6acd830d360b83250c630
13:M 22 Jun 2022 14:27:35.561 * Running mode=cluster, port=6379.
13:M 22 Jun 2022 14:27:35.561 # Server initialized
13:M 22 Jun 2022 14:27:35.561 * Ready to accept connections
13:S 22 Jun 2022 14:28:57.588 * Before turning into a replica, using my own master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer.
13:S 22 Jun 2022 14:28:57.588 * Connecting to MASTER 172.16.84.122:6379
13:S 22 Jun 2022 14:28:57.588 * MASTER <-> REPLICA sync started
13:S 22 Jun 2022 14:28:57.588 # Cluster state changed: ok
13:S 22 Jun 2022 14:28:57.588 * Non blocking connect for SYNC fired the event.
13:S 22 Jun 2022 14:28:57.589 * Master replied to PING, replication can continue...
13:S 22 Jun 2022 14:28:57.589 * Trying a partial resynchronization (request 8e3ffd79b9f7370b05effda9a837c2956e5fabed:1).
13:S 22 Jun 2022 14:28:57.590 * Full resync from master: 8e2dab8037c5e9c194cf0c659c9c28f77c08614e:0
13:S 22 Jun 2022 14:28:57.590 * Discarding previously cached master state.
13:S 22 Jun 2022 14:28:57.619 * MASTER <-> REPLICA sync: receiving 175 bytes from master to disk
13:S 22 Jun 2022 14:28:57.619 * MASTER <-> REPLICA sync: Flushing old data
13:S 22 Jun 2022 14:28:57.619 * MASTER <-> REPLICA sync: Loading DB in memory
13:S 22 Jun 2022 14:28:57.620 * Loading RDB produced by version 6.2.5
13:S 22 Jun 2022 14:28:57.620 * RDB age 0 seconds
13:S 22 Jun 2022 14:28:57.620 * RDB memory usage when created 2.60 Mb
13:S 22 Jun 2022 14:28:57.620 * MASTER <-> REPLICA sync: Finished with success
13:S 22 Jun 2022 14:28:57.621 * Background append only file rewriting started by pid 83
13:S 22 Jun 2022 14:28:57.650 * AOF rewrite child asks to stop sending diffs.
83:C 22 Jun 2022 14:28:57.650 * Parent agreed to stop sending diffs. Finalizing AOF...
83:C 22 Jun 2022 14:28:57.650 * Concatenating 0.00 MB of AOF diff received from parent.
83:C 22 Jun 2022 14:28:57.651 * SYNC append only file rewrite performed
83:C 22 Jun 2022 14:28:57.651 * AOF rewrite: 0 MB of memory used by copy-on-write
13:S 22 Jun 2022 14:28:57.725 * Background AOF rewrite terminated with success
13:S 22 Jun 2022 14:28:57.725 * Residual parent diff successfully flushed to the rewritten AOF (0.00 MB)
13:S 22 Jun 2022 14:28:57.725 * Background AOF rewrite finished successfully

이제 failover(장애 조치) 테스트를 진행 해볼텐데 그전에 pods의 IP를 기억해두시기 바랍니다.

$ kubectl get pod -n redis-system -o wide
NAME                       READY   STATUS    RESTARTS      AGE   IP              NODE            NOMINATED NODE   READINESS GATES
redis-cluster-follower-0   2/2     Running   0             1h    172.16.84.121   k8s-w3-dragon   <none>           <none>
redis-cluster-follower-1   2/2     Running   0             1h    172.16.10.75    k8s-w2-dragon   <none>           <none>
redis-cluster-follower-2   2/2     Running   0             1h    172.16.84.115   k8s-w3-dragon   <none>           <none>
redis-cluster-leader-0     2/2     Running   0             1h    172.16.84.101   k8s-w3-dragon   <none>           <none>
redis-cluster-leader-1     2/2     Running   0             1h    172.16.10.103   k8s-w2-dragon   <none>           <none>
redis-cluster-leader-2     2/2     Running   0             1h    172.16.78.67    k8s-w1-dragon   <none>           <none>

Master nodes
- master 1 (redis-cluster-leader-0) : 172.16.84.101
- master 2 (redis-cluster-leader-1) : 172.16.10.103
- master 3 (redis-cluster-leader-2) : 172.16.78.67
Replica nodes
- replica 1 (redis-cluster-follower-0) : 172.16.84.121
- replica 2 (redis-cluster-follower-1) : 172.16.10.78
- replica 3 (redis-cluster-follower-2) : 172.16.84.115

이제 1번 master node (redis-cluster-leader-0 pod)의 image tag를 존재하지 않는 것으로 변경하여 임으로 장애를 발생시키겠습니다.

$ kubectl patch pod/redis-cluster-leader-0 -n redis-system -p '{"spec":{"containers":[{"name": "redis-cluster-leader","image": "quay.io/opstree/redis:v6.2.9"}]}}'

[이미지 출처] http://redisgate.kr/redis/cluster/cluster_failover.php

image tag를 변경후 1번 master node(redis-cluster-leader-0 pod)의 상태가 ImagePullBackOff 상태로 변경되었습니다.

$ kubectl get pod -n redis-system -o wide
NAME                       READY   STATUS             RESTARTS      AGE     IP              NODE            NOMINATED NODE   READINESS GATES
redis-cluster-follower-0   2/2     Running            0             15h     172.16.84.121   k8s-w3-dragon   <none>           <none>
redis-cluster-follower-1   2/2     Running            0             15h     172.16.10.75    k8s-w2-dragon   <none>           <none>
redis-cluster-follower-2   2/2     Running            0             15h     172.16.84.115   k8s-w3-dragon   <none>           <none>
redis-cluster-leader-0     1/2     ImagePullBackOff   0 (47s ago)   2m21s   172.16.84.101   k8s-w3-dragon   <none>           <none>
redis-cluster-leader-1     2/2     Running            0             15h     172.16.10.103   k8s-w2-dragon   <none>           <none>
redis-cluster-leader-2     2/2     Running            0             15h     172.16.78.67    k8s-w1-dragon   <none>           <none>

1번 master node(redis-cluster-leader-0 pod) : 172.16.84.101 에서 redis cluster nodes 상태를 확인하면, node fail 상태로 연결이 끊어졌습니다.

그리고 ***1번 replica node(redis-cluster-follower-0 pod) : 172.16.84.121***가 master node로 leader 선출에 따라 승격이 되었습니다.

$ kubectl exec -it redis-cluster-leader-1 -n redis-system -c redis-cluster-leader -- redis-cli -a redispassword cluster nodes
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
5483f9be46324b9e8f48b431fc4128c788525cdc 172.16.10.103:6379@16379 myself,master - 0 1655964724000 2 connected 5461-10922
742149aba5d2e572b6f85ea59a964880fa79b367 172.16.78.67:6379@16379 master - 0 1655964724000 3 connected 10923-16383
a956b4aaf4a04957bc6b0a42bdb62122c91fb44e 172.16.84.115:6379@16379 slave 742149aba5d2e572b6f85ea59a964880fa79b367 0 1655964724864 3 connected
4d0bd6e99c00cddd6f9858da371526f39fef157a 172.16.10.75:6379@16379 slave 5483f9be46324b9e8f48b431fc4128c788525cdc 0 1655964725573 2 connected
87fe6e67e43ddb9f9ad6acd830d360b83250c630 ***172.16.84.121:6379@16379 master*** - 0 1655964725000 4 connected 0-5460
7ded434ed7be8db05a7851dc44e082e38bbf398b ***172.16.84.101:6379@16379 master,fail*** - 1655964668967 1655964667000 1 ***disconnected***

master로 승격된 ***1번 replica node(redis-cluster-follower-0 pod) : 172.16.84.121***의 redis log를 출력하면 Failover election에 따라 자기가 master가 되었다는 것을 알립니다.

Failover election won: I'm the new master.

$ kubectl logs -f redis-cluster-follower-0 -n redis-system
.
.
.
13:S 23 Jun 2022 06:11:14.473 * Connecting to MASTER 172.16.84.101:6379
13:S 23 Jun 2022 06:11:14.473 * MASTER <-> REPLICA sync started
13:S 23 Jun 2022 06:11:14.473 # Error condition on socket for SYNC: Connection refused
13:S 23 Jun 2022 06:11:14.607 * FAIL message received from 742149aba5d2e572b6f85ea59a964880fa79b367 about 7ded434ed7be8db05a7851dc44e082e38bbf398b
13:S 23 Jun 2022 06:11:14.677 # Start of election delayed for 510 milliseconds (rank #0, offset 126).
13:S 23 Jun 2022 06:11:15.283 # Starting a failover election for epoch 4.
***13:S 23 Jun 2022 06:11:15.287 # Failover election won: I'm the new master.***
13:S 23 Jun 2022 06:11:15.287 # configEpoch set to 4 after successful failover
13:M 23 Jun 2022 06:11:15.287 * Discarding previously cached master state.
13:M 23 Jun 2022 06:11:15.287 # Setting secondary replication ID to 26d81a3e61a5425bfbab71716df5092c0d6deda8, valid up to offset: 127. New replication ID is 995090ede0799f7b0f13e5069b8ac40566902b0b

이제 1번 master node는 redis-cluster-leader-0 pod : 172.16.84.101에서 redis-cluster-follower-0 pod : 172.16.84.121로 변경 되었습니다.

그러면 이제 임의로 장애를 발생시킨 redis-cluster-leader-0 pod를 복원하면 어떻게 될까요?

master로 복원(승격)된다.
replica로 복원된다.
아무런 영향도 없다.

한번 복원하여 상태를 살펴보도록 하겠습니다. image tag를 원래의 v6.2.5로 변경합니다.

$ kubectl patch pod/redis-cluster-leader-0 -n redis-system -p '{"spec":{"containers":[{"name": "redis-cluster-leader","image": "quay.io/opstree/redis:v6.2.5"}]}}'

cluster_master_down_2.png

pod의 상태가 복원 되었는지 확인합니다. 잘 복원 되었네요.

$ kubectl get pod -n redis-system -o wide
NAME                       READY   STATUS    RESTARTS      AGE   IP              NODE            NOMINATED NODE   READINESS GATES
redis-cluster-follower-0   2/2     Running   0             1h    172.16.84.121   k8s-w3-dragon   <none>           <none>
redis-cluster-follower-1   2/2     Running   0             1h    172.16.10.75    k8s-w2-dragon   <none>           <none>
redis-cluster-follower-2   2/2     Running   0             1h    172.16.84.115   k8s-w3-dragon   <none>           <none>
redis-cluster-leader-0     2/2     Running   1 (3m ago)    3m    172.16.84.101   k8s-w3-dragon   <none>           <none>
redis-cluster-leader-1     2/2     Running   0             1h    172.16.10.103   k8s-w2-dragon   <none>           <none>
redis-cluster-leader-2     2/2     Running   0             1h    172.16.78.67    k8s-w1-dragon   <none>           <none>

redis cluster nodes의 상태를 확인해봅니다.

$ kubectl exec -it redis-cluster-leader-0 -n redis-system -c redis-cluster-leader -- redis-cli -a redispassword cluster nodes
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
5483f9be46324b9e8f48b431fc4128c788525cdc 172.16.10.103:6379@16379 master - 0 1655966203034 2 connected 5461-10922
7ded434ed7be8db05a7851dc44e082e38bbf398b 172.16.84.101:6379@16379 myself,slave 87fe6e67e43ddb9f9ad6acd830d360b83250c630 0 1655966203000 4 connected
87fe6e67e43ddb9f9ad6acd830d360b83250c630 172.16.84.121:6379@16379 master - 0 1655966202018 4 connected 0-5460
742149aba5d2e572b6f85ea59a964880fa79b367 172.16.78.67:6379@16379 master - 0 1655966204047 3 connected 10923-16383
a956b4aaf4a04957bc6b0a42bdb62122c91fb44e 172.16.84.115:6379@16379 slave 742149aba5d2e572b6f85ea59a964880fa79b367 0 1655966203000 3 connected
4d0bd6e99c00cddd6f9858da371526f39fef157a 172.16.10.75:6379@16379 slave 5483f9be46324b9e8f48b431fc4128c788525cdc 0 1655966203541 2 connected

기존 node를 복원하더라도 master로 복원되는 것이 아닌 replica로 복원되는 것을 알 수 있습니다.

즉 master가 장애후 replica가 master로 승격된 이후에는 장애가 났던 master를 복원하더라도 이미 failover election에 따라 replica가 master로 승격이 되어버린 상태에서는 복원된 node는 replica의 역할이 된다는 것을 알 수있습니다.

마지막으로 아래는 redis-cluster-follower-0과 redis-cluster-leader-0의 redis logs 입니다.

$ kubectl logs -f redis-cluster-follower-0 -n redis-system -c redis-cluster-follower
.
.
.
13:M 23 Jun 2022 06:35:08.883 * Clear FAIL state for node 7ded434ed7be8db05a7851dc44e082e38bbf398b: master without slots is reachable again.
13:M 23 Jun 2022 06:35:08.885 * Replica 172.16.84.101:6379 asks for synchronization
13:M 23 Jun 2022 06:35:08.885 * Partial resynchronization not accepted: Replication ID mismatch (Replica asked for '6d26e33239942b867bb6df23cee2c2171d556b0b', my replication IDs are '995090ede0799f7b0f13e5069b8ac40566902b0b' and '26d81a3e61a5425bfbab71716df5092c0d6deda8')
13:M 23 Jun 2022 06:35:08.885 * Starting BGSAVE for SYNC with target: disk
13:M 23 Jun 2022 06:35:08.901 * Background saving started by pid 61429
61429:C 23 Jun 2022 06:35:08.903 * DB saved on disk
61429:C 23 Jun 2022 06:35:08.903 * RDB: 0 MB of memory used by copy-on-write
13:M 23 Jun 2022 06:35:08.982 * Background saving terminated with success
13:M 23 Jun 2022 06:35:08.982 * Synchronization with replica 172.16.84.101:6379 succeeded

$ kubectl logs -f redis-cluster-leader-0 -n redis-system -c redis-cluster-leader
Starting redis service in cluster mode.....
11:C 23 Jun 2022 06:35:08.878 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
11:C 23 Jun 2022 06:35:08.878 # Redis version=6.2.5, bits=64, commit=00000000, modified=0, pid=11, just started
11:C 23 Jun 2022 06:35:08.878 # Configuration loaded
11:M 23 Jun 2022 06:35:08.879 * monotonic clock: POSIX clock_gettime
11:M 23 Jun 2022 06:35:08.879 * Node configuration loaded, I'm 7ded434ed7be8db05a7851dc44e082e38bbf398b
11:M 23 Jun 2022 06:35:08.879 * Running mode=cluster, port=6379.
11:M 23 Jun 2022 06:35:08.879 # Server initialized
11:M 23 Jun 2022 06:35:08.881 * DB loaded from append only file: 0.000 seconds
11:M 23 Jun 2022 06:35:08.881 * Ready to accept connections
11:M 23 Jun 2022 06:35:08.882 # Configuration change detected. Reconfiguring myself as a replica of 87fe6e67e43ddb9f9ad6acd830d360b83250c630
11:S 23 Jun 2022 06:35:08.882 * Before turning into a replica, using my own master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer.
11:S 23 Jun 2022 06:35:08.882 * Connecting to MASTER 172.16.84.121:6379
11:S 23 Jun 2022 06:35:08.882 * MASTER <-> REPLICA sync started
11:S 23 Jun 2022 06:35:08.882 # Cluster state changed: ok
11:S 23 Jun 2022 06:35:08.883 * Non blocking connect for SYNC fired the event.
11:S 23 Jun 2022 06:35:08.884 * Master replied to PING, replication can continue...
11:S 23 Jun 2022 06:35:08.884 * Trying a partial resynchronization (request 6d26e33239942b867bb6df23cee2c2171d556b0b:1).
11:S 23 Jun 2022 06:35:08.901 * Full resync from master: 995090ede0799f7b0f13e5069b8ac40566902b0b:126
11:S 23 Jun 2022 06:35:08.901 * Discarding previously cached master state.
11:S 23 Jun 2022 06:35:08.982 * MASTER <-> REPLICA sync: receiving 194 bytes from master to disk
11:S 23 Jun 2022 06:35:08.982 * MASTER <-> REPLICA sync: Flushing old data
11:S 23 Jun 2022 06:35:08.983 * MASTER <-> REPLICA sync: Loading DB in memory
11:S 23 Jun 2022 06:35:08.983 * Loading RDB produced by version 6.2.5
11:S 23 Jun 2022 06:35:08.983 * RDB age 0 seconds
11:S 23 Jun 2022 06:35:08.984 * RDB memory usage when created 2.54 Mb
11:S 23 Jun 2022 06:35:08.984 * MASTER <-> REPLICA sync: Finished with success
11:S 23 Jun 2022 06:35:08.984 * Background append only file rewriting started by pid 16
11:S 23 Jun 2022 06:35:09.026 * AOF rewrite child asks to stop sending diffs.
16:C 23 Jun 2022 06:35:09.026 * Parent agreed to stop sending diffs. Finalizing AOF...
16:C 23 Jun 2022 06:35:09.026 * Concatenating 0.00 MB of AOF diff received from parent.
16:C 23 Jun 2022 06:35:09.026 * SYNC append only file rewrite performed
16:C 23 Jun 2022 06:35:09.026 * AOF rewrite: 0 MB of memory used by copy-on-write
11:S 23 Jun 2022 06:35:09.083 * Background AOF rewrite terminated with success
11:S 23 Jun 2022 06:35:09.083 * Residual parent diff successfully flushed to the rewritten AOF (0.00 MB)
11:S 23 Jun 2022 06:35:09.083 * Background AOF rewrite finished successfully
11:S 23 Jun 2022 06:50:09.009 * 1 changes in 900 seconds. Saving...
11:S 23 Jun 2022 06:50:09.010 * Background saving started by pid 969
969:C 23 Jun 2022 06:50:09.011 * DB saved on disk
969:C 23 Jun 2022 06:50:09.012 * RDB: 0 MB of memory used by copy-on-write
11:S 23 Jun 2022 06:50:09.111 * Background saving terminated with success

Step 5. Redis Cluster TCP ports

모든 Redis Cluster는 두개의 TCP connection을 필요로 하며, Cluster를 구성하는 각 Redis instances는 다른 모든 Redis instance과 Gassip Protocol을 사용하여 redis 상태 정보를 교환합니다.

기본적으로 Redis TCP Port(6379)는 Client를 위해 사용되며, 여기에 10000을 더해 Redis instance(node)간에 통신을 위한 Cluster Bus Port(16379)를 추가로 생성합니다.

Master 노드가 다운되면 gassip protocol을 이용해 redis 상태를 확인하고 replica중 하나를 master로 승격시키는데 사용됩니다.

Cluster Bus Port

Cluster Bus Port는 node-to-node 통신 채널로 사용
failure detection, configuration update, failover authorization 등에 사용
Clint는 Redis TCP Port(6379)를 사용하면 되지만, Redis Cluster의 node-to-node 통신을 위해 Cluster Bus Port(16379)를 방화벽에서 허용 필요

[이미지 출처] https://alibaba-cloud.medium.com/understanding-the-failover-mechanism-of-redis-cluster-3b979dcf3441

Official Website

Website
- https://ot-container-kit.github.io/redis-operator/
- Legacy Website [링크]
- example
  - https://github.com/OT-CONTAINER-KIT/redis-operator/tree/master/example
GitHub
- https://github.com/OT-CONTAINER-KIT/redis-operator
- API Spec
  - redis_types.go [링크]
  - rediscluster_types.go [링크]
  - common_types.go [링크]
Artifact Hub
- https://artifacthub.io/packages/olm/community-operators/redis-operator
Helm chart repository
- Redis Operator [링크]
- Redis Standalone [링크]
- Redis Cluster [링크]

Reference

Redis Cluster 101 [링크]
Redis Cluster 사용하기와 Redis Cluster Proxy 필요성 [링크]
Redis Cluster의 Failover Mechanism 이해 [링크]
Redis Cluster 과반 이상 Master shutdown 되었을때 대응 방법 [링크]
Redis Cluster 장애 복구 시뮬레이션 [링크]
Redis Cluster Failover 장애복구 [링크]
Redis Cluster nodes.conf 파일을 이용한 장애복구 [링크]
Gassip protocol wiki [링크]
Gassip 프로토콜이란? [링크]

END