Kubernetes生產實踐之Redis-cluster
作者：老實說運維
在Kubernetes中部署Redis集群面臨挑戰，因為每個Redis實例都依賴于一個配置文件，該文件可以跟蹤其他集群實例及其角色。為此，需要結合使用Kubernetes StatefulSets和PersistentVolumes實現。
方案一 自定義yaml文件安裝redis cluster
背景
在Kubernetes中部署Redis集群面臨挑戰，因為每個Redis實例都依賴于一個配置文件，該文件可以跟蹤其他集群實例及其角色。為此，需要結合使用Kubernetes StatefulSets和PersistentVolumes實現。
redis cluster架構示意圖：
創建StatefulSet yaml文件
- ---
- apiVersion: v1
- kind: ConfigMap
- metadata:
- name: redis-cluster
- data:
- update.sh: |
- #!/bin/sh
- REDIS_NODES="/data/nodes.conf"
- sed -i -e "/myself/ s/[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}/${POD_IP}/" ${REDIS_NODES}
- exec "$@"
- redis.conf: |+
- bind 0.0.0.0
- cluster-enabled yes
- cluster-require-full-coverage no
- cluster-node-timeout 30000
- cluster-config-file /data/nodes.conf
- cluster-migration-barrier 1
- appendonly yes
- protected-mode no
- ---
- apiVersion: apps.kruise.io/v1beta1
- # apiVersion: apps/v1
- kind: StatefulSet
- metadata:
- name: redis-cluster
- spec:
- serviceName: redis-cluster
- replicas: 6
- selector:
- matchLabels:
- app: redis-cluster
- template:
- metadata:
- labels:
- app: redis-cluster
- spec:
- containers:
- - name: redis
- image: redis:6.2.1-alpine
- ports:
- - containerPort: 6379
- name: client
- - containerPort: 16379
- name: gossip
- command: ["/conf/update.sh", "redis-server", "/conf/redis.conf"]
- env:
- - name: POD_IP
- valueFrom:
- fieldRef:
- fieldPath: status.podIP
- volumeMounts:
- - name: conf
- mountPath: /conf
- readOnly: false
- - name: data
- mountPath: /data
- readOnly: false
- volumes:
- - name: conf
- configMap:
- name: redis-cluster
- defaultMode: 0755
- volumeClaimTemplates:
- - metadata:
- name: data
- spec:
- accessModes: [ "ReadWriteOnce" ]
- resources:
- requests:
- storage: 100Gi
- storageClassName: rbd
注釋：
- cluster-migration-barrier 那些分配后仍然剩余migration barrier個從節點的主節點才會觸發節點分配，而不是分配前有migration barrier個從節點的主節點就會觸發節點分配，默認是1，生產環境建議維持默認值
- protected-mode no 參數是為了禁止外網訪問redis，如果啟用了，則只能夠通過lookback ip（127.0.0.1）訪問Redis，如果外網訪問redis，會報出異常。
- apiVersion: apps.kruise.io/v1beta1 控制器這里使用kruise提供的Advanced StatefulSet，如果集群沒有安裝kruise，可以使用 apps/v1
安裝redis-cluster
- [root@qd01-stop-k8s-master001 redis]# kubectl apply -f install-redis.yaml
- configmap/redis-cluster created
- statefulset.apps.kruise.io/redis-cluster created
- [root@qd01-stop-k8s-master001 redis]# kubectl get po -n op
- NAME READY STATUS RESTARTS AGE
- redis-cluster-0 1/1 Running 0 3m26s
- redis-cluster-1 1/1 Running 0 3m14s
- redis-cluster-2 1/1 Running 0 2m54s
- redis-cluster-3 1/1 Running 0 2m23s
- redis-cluster-4 1/1 Running 0 2m14s
- redis-cluster-5 1/1 Running 0 114s
創建redis-cluster service
- ---
- apiVersion: v1
- kind: Service
- metadata:
- name: redis-cluster
- namespace: op
- spec:
- type: ClusterIP
- ports:
- - port: 6379
- targetPort: 6379
- name: client
- - port: 16379
- targetPort: 16379
- name: gossip
- selector:
- app: redis-cluster
- [root@qd01-stop-k8s-master001 redis]# kubectl apply -f redis-svc.yml
- service/redis-cluster created
- [root@qd01-stop-k8s-master001 redis]# kubectl get svc -n op
- NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
- redis-cluster ClusterIP 10.97.197.224 <none> 6379/TCP,16379/TCP 9s
- 測試能后連通
- [root@qd01-stop-k8s-master001 redis]# telnet 10.97.197.224 6379
- Trying 10.97.197.224...
- Connected to 10.97.197.224.
- Escape character is '^]'.
初始化redis-cluster
執行如下命令，獲取到pod IP，然后使用redis-cli --cluster創建集群
- [root@qd01-stop-k8s-master001 redis]# kubectl -n op exec -it redis-cluster-0 -- redis-cli --cluster create --cluster-replicas 1 $(kubectl -n op get pods -l app=redis-cluster -o jsonpath='{range.items[*]}{.status.podIP}:6379 {end}')
- >>> Performing hash slots allocation on 6 nodes...
- Master[0] -> Slots 0 - 5460
- Master[1] -> Slots 5461 - 10922
- Master[2] -> Slots 10923 - 16383
- Adding replica 100.88.43.67:6379 to 100.64.147.152:6379
- Adding replica 100.113.170.5:6379 to 100.98.174.217:6379
- Adding replica 100.64.147.153:6379 to 100.80.158.227:6379
- M: b47b27a3dbddf3fc1370cbe14ae753f4fce20b04 100.64.147.152:6379
- slots:[0-5460] (5461 slots) master
- M: 09543217c903350e963fc4fdf4acb73f8a1b7f8b 100.98.174.217:6379
- slots:[5461-10922] (5462 slots) master
- M: 5389ace495b68eeac85370d6783648dff68f2fb6 100.80.158.227:6379
- slots:[10923-16383] (5461 slots) master
- S: b1f39714c006ae55b12b18e6537303d7a00e1704 100.64.147.153:6379
- replicates 5389ace495b68eeac85370d6783648dff68f2fb6
- S: 0113f4668ec2f3ca2e9470c44bd5faab532b0936 100.88.43.67:6379
- replicates b47b27a3dbddf3fc1370cbe14ae753f4fce20b04
- S: e1e2f18ae66c79f1943390beabb59613abbad38a 100.113.170.5:6379
- replicates 09543217c903350e963fc4fdf4acb73f8a1b7f8b
- Can I set the above configuration? (type 'yes' to accept): yes
- >>> Nodes configuration updated
- >>> Assign a different config epoch to each node
- >>> Sending CLUSTER MEET messages to join the cluster
- Waiting for the cluster to join
- ..
- >>> Performing Cluster Check (using node 100.64.147.152:6379)
- M: b47b27a3dbddf3fc1370cbe14ae753f4fce20b04 100.64.147.152:6379
- slots:[0-5460] (5461 slots) master
- 1 additional replica(s)
- S: 0113f4668ec2f3ca2e9470c44bd5faab532b0936 100.88.43.67:6379
- slots: (0 slots) slave
- replicates b47b27a3dbddf3fc1370cbe14ae753f4fce20b04
- M: 09543217c903350e963fc4fdf4acb73f8a1b7f8b 100.98.174.217:6379
- slots:[5461-10922] (5462 slots) master
- 1 additional replica(s)
- M: 5389ace495b68eeac85370d6783648dff68f2fb6 100.80.158.227:6379
- slots:[10923-16383] (5461 slots) master
- 1 additional replica(s)
- S: e1e2f18ae66c79f1943390beabb59613abbad38a 100.113.170.5:6379
- slots: (0 slots) slave
- replicates 09543217c903350e963fc4fdf4acb73f8a1b7f8b
- S: b1f39714c006ae55b12b18e6537303d7a00e1704 100.64.147.153:6379
- slots: (0 slots) slave
- replicates 5389ace495b68eeac85370d6783648dff68f2fb6
- [OK] All nodes agree about slots configuration.
- >>> Check for open slots...
- >>> Check slots coverage...
- [OK] All 16384 slots covered.
驗證集群信息
- [root@qd01-stop-k8s-master001 redis]# kubectl -n op exec -it redis-cluster-0 -- redis-cli cluster info
- cluster_state:ok
- cluster_slots_assigned:16384
- cluster_slots_ok:16384
- cluster_slots_pfail:0
- cluster_slots_fail:0
- cluster_known_nodes:6
- cluster_size:3
- cluster_current_epoch:6
- cluster_my_epoch:1
- cluster_stats_messages_ping_sent:178
- cluster_stats_messages_pong_sent:181
- cluster_stats_messages_sent:359
- cluster_stats_messages_ping_received:176
- cluster_stats_messages_pong_received:178
- cluster_stats_messages_meet_received:5
- cluster_stats_messages_received:359
- [root@qd01-stop-k8s-master001 redis]# kubectl -n op exec -it redis-cluster-0 -- redis-cli cluster nodes
- 0113f4668ec2f3ca2e9470c44bd5faab532b0936 100.88.43.67:6379@16379 slave b47b27a3dbddf3fc1370cbe14ae753f4fce20b04 0 1615348311156 1 connected
- 09543217c903350e963fc4fdf4acb73f8a1b7f8b 100.98.174.217:6379@16379 master - 0 1615348314162 2 connected 5461-10922
- b47b27a3dbddf3fc1370cbe14ae753f4fce20b04 100.64.147.152:6379@16379 myself,master - 0 1615348312000 1 connected 0-5460
- 5389ace495b68eeac85370d6783648dff68f2fb6 100.80.158.227:6379@16379 master - 0 1615348312000 3 connected 10923-16383
- e1e2f18ae66c79f1943390beabb59613abbad38a 100.113.170.5:6379@16379 slave 09543217c903350e963fc4fdf4acb73f8a1b7f8b 0 1615348313160 2 connected
- b1f39714c006ae55b12b18e6537303d7a00e1704 100.64.147.153:6379@16379 slave 5389ace495b68eeac85370d6783648dff68f2fb6 0 1615348312158 3 connected
從輸出可以看到，集群總共6個節點，三主三從
方案二 使用kubeDB安裝redis
安裝kubeDB
1、安裝KubeDB
獲取AppsCode License
https://license-issuer.appscode.com/
下載KubeDB charts
https://github.com/appscode/charts/tree/master/stable/kubedb-community
- [root@qd01-stop-k8s-master001 kubedb-community]# unzip kubedb-community-v0.16.2.tgz
- [root@qd01-stop-k8s-master001 kubedb-community]# cd kubedb-community
- [root@qd01-stop-k8s-master001 kubedb-community]# ls -al
- total 96
- drwxr-xr-x 4 root root 158 Mar 10 15:26 .
- drwxr-xr-x 3 root root 66 Mar 10 15:24 ..
- -rw-r--r-- 1 root root 351 Feb 16 09:55 Chart.yaml
- drwxr-xr-x 2 root root 28 Mar 10 15:24 ci
- -rw-r--r-- 1 root root 493 Feb 16 09:55 doc.yaml
- -rw-r--r-- 1 root root 353 Feb 16 09:55 .helmignore
- -rw-r--r-- 1 root root 24422 Feb 16 09:55 README.md
- drwxr-xr-x 2 root root 4096 Mar 10 15:24 templates
- -rw-r--r-- 1 root root 47437 Feb 16 09:55 values.openapiv3_schema.yaml
- -rw-r--r-- 1 root root 5230 Feb 16 09:55 values.yaml
修改values.yaml，把License文件放到kubedb-community目錄下
2、使用helm安裝
- [root@qd01-stop-k8s-master001 kubedb-community]# helm install kubedb-community --namespace kube-system --set-file license=./kubedb-community-license.txt -f values.yaml .
- NAME: kubedb-community
- LAST DEPLOYED: Wed Mar 10 15:38:59 2021
- NAMESPACE: kube-system
- STATUS: deployed
- REVISION: 1
- TEST SUITE: None
- NOTES:
- To verify that KubeDB has started, run:
- kubectl get deployment --namespace kube-system -l "app.kubernetes.io/name=kubedb-community,app.kubernetes.io/instance=kubedb-community"
- Now install/upgrade appscode/kubedb-catalog chart.
- To install, run:
- helm install kubedb-catalog appscode/kubedb-catalog --version v0.16.2 --namespace kube-system
- To upgrade, run:
- helm upgrade kubedb-catalog appscode/kubedb-catalog --version v0.16.2 --namespace kube-system
- 執行如下命令查看是否安裝完成
- [root@qd01-stop-k8s-master001 kubedb-community]# kubectl get deployment --namespace kube-system -l "app.kubernetes.io/name=kubedb-community,app.kubernetes.io/instance=kubedb-community"
- NAME READY UP-TO-DATE AVAILABLE AGE
- kubedb-community 1/1 1 1 38s
- 等待crds注冊成功
- [root@qd01-stop-k8s-master001 kubedb-community]# kubectl get crds -l app.kubernetes.io/name=kubedb -w
- NAME CREATED AT
- elasticsearches.kubedb.com 2021-03-10T07:39:42Z
- elasticsearchversions.catalog.kubedb.com 2021-03-10T07:39:45Z
- etcds.kubedb.com 2021-03-10T07:39:42Z
- etcdversions.catalog.kubedb.com 2021-03-10T07:39:45Z
- memcacheds.kubedb.com 2021-03-10T07:39:43Z
- memcachedversions.catalog.kubedb.com 2021-03-10T07:39:45Z
- mongodbs.kubedb.com 2021-03-10T07:39:43Z
- mongodbversions.catalog.kubedb.com 2021-03-10T07:39:45Z
- mysqls.kubedb.com 2021-03-10T07:39:43Z
- mysqlversions.catalog.kubedb.com 2021-03-10T07:39:46Z
- perconaxtradbs.kubedb.com 2021-03-10T07:39:43Z
- perconaxtradbversions.catalog.kubedb.com 2021-03-10T07:39:46Z
- pgbouncers.kubedb.com 2021-03-10T07:39:44Z
- pgbouncerversions.catalog.kubedb.com 2021-03-10T07:39:46Z
- postgreses.kubedb.com 2021-03-10T07:39:44Z
- postgresversions.catalog.kubedb.com 2021-03-10T07:39:46Z
- proxysqls.kubedb.com 2021-03-10T07:39:44Z
- proxysqlversions.catalog.kubedb.com 2021-03-10T07:39:46Z
- redises.kubedb.com 2021-03-10T07:39:45Z
- redisversions.catalog.kubedb.com 2021-03-10T07:39:46Z
3、安裝KubeDB Catalog
同樣，先下載
https://github.com/appscode/charts/tree/master/stable/kubedb-catalog
- [root@qd01-stop-k8s-master001 kubedb-catalog]# tar -zxf kubedb-catalog-v0.16.2.tgz
- [root@qd01-stop-k8s-master001 kubedb-catalog]# cd kubedb-catalog
- [root@qd01-stop-k8s-master001 kubedb-catalog]# ls -al
- total 24
- drwxr-xr-x 3 root root 148 Mar 10 15:48 .
- drwxr-xr-x 3 root root 28 Mar 10 15:48 ..
- -rw-r--r-- 1 root root 321 Jan 26 20:08 Chart.yaml
- -rw-r--r-- 1 root root 467 Jan 26 20:08 doc.yaml
- -rw-r--r-- 1 root root 353 Jan 26 20:08 .helmignore
- -rw-r--r-- 1 root root 3195 Jan 26 20:08 README.md
- drwxr-xr-x 12 root root 188 Mar 10 15:48 templates
- -rw-r--r-- 1 root root 744 Jan 26 20:08 values.openapiv3_schema.yaml
- -rw-r--r-- 1 root root 1070 Jan 26 20:08 values.yaml
- [root@qd01-stop-k8s-master001 kubedb-catalog]# helm install kubedb-catalog --namespace kube-system -f values.yaml .
- NAME: kubedb-catalog
- LAST DEPLOYED: Wed Mar 10 15:50:50 2021
- NAMESPACE: kube-system
- STATUS: deployed
- REVISION: 1
- TEST SUITE: None
使用kubedb安裝redis
1、先看下官方給的redis生命周期示意圖
kubedb安裝redis支持如下特性
- Features Availability
- Clustering ✓
- Instant Backup ✗
- Scheduled Backup ✗
- Persistent Volume ✓
- Initialize using Snapshot ✗
- Initialize using Script ✗
- Custom Configuration ✓
- Using Custom docker image ✓
- Builtin Prometheus Discovery ✓
- Using Prometheus operator ✓
2、查看支持的版本
- [root@qd01-stop-k8s-master001 kubedb-catalog]# kubectl get redisversions
- NAME VERSION DB_IMAGE DEPRECATED AGE
- 4.0.11 4.0.11 kubedb/redis:4.0.11 15m
- 4.0.6-v2 4.0.6 kubedb/redis:4.0.6-v2 15m
- 5.0.3-v1 5.0.3 kubedb/redis:5.0.3-v1 15m
- 6.0.6 6.0.6 kubedb/redis:6.0.6 15m
3、編輯yaml安裝文件
可以參照
https://github.com/kubedb/docs/blob/v2021.01.26/docs/examples/redis/clustering/demo-1.yaml
這里選擇安裝6.0.6這個版本，我的集群storageClassName: "rbd"，請根據實際修改
如果想自定義redis.conf,請參考
https://github.com/kubedb/docs/blob/v2021.01.26/docs/examples/redis/custom-config/redis-custom.yaml
- apiVersion: kubedb.com/v1alpha2
- kind: Redis
- metadata:
- name: redis-cluster
- namespace: op
- spec:
- version: 6.0.6
- mode: Cluster
- cluster:
- master: 3
- replicas: 1
- storageType: Durable
- storage:
- resources:
- requests:
- storage: 1Gi
- storageClassName: "rbd"
- accessModes:
- - ReadWriteOnce
執行安裝
- [root@qd01-stop-k8s-master001 kubedb-community]# kubectl apply -f redis-cluster.yaml
- redis.kubedb.com/redis-cluster created
- 安裝完成，可以如下查看
- [root@qd01-stop-k8s-master001 kubedb-community]# kubectl get rd,po -n op
- NAME VERSION STATUS AGE
- redis.kubedb.com/redis-cluster 6.0.6 Provisioning 6m55s
- NAME READY STATUS RESTARTS AGE
- pod/redis-cluster-shard0-0 1/1 Running 0 6m54s
- pod/redis-cluster-shard0-1 1/1 Running 0 6m18s
- pod/redis-cluster-shard1-0 1/1 Running 0 5m38s
- pod/redis-cluster-shard1-1 1/1 Running 0 5m1s
- pod/redis-cluster-shard2-0 1/1 Running 0 4m30s
- pod/redis-cluster-shard2-1 1/1 Running 0 4m8s
- [root@qd01-stop-k8s-master001 redis]# kubectl get svc -n op
- NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
- redis-cluster ClusterIP 10.97.197.224 <none> 6379/TCP 5h16m
- redis-cluster-pods ClusterIP None <none> 6379/TCP 17m
4、驗證集群
- [root@qd01-stop-k8s-master001 kubedb-community]# kubectl get pods -n op -o jsonpath='{range.items[*]}{.metadata.name} ---------- {.status.podIP}:6379{"\t\n"}{end}' | grep redis
- redis-cluster-shard0-0 ---------- 100.64.147.156:6379
- redis-cluster-shard0-1 ---------- 100.98.174.218:6379
- redis-cluster-shard1-0 ---------- 100.126.252.204:6379
- redis-cluster-shard1-1 ---------- 100.113.170.6:6379
- redis-cluster-shard2-0 ---------- 100.107.55.69:6379
- redis-cluster-shard2-1 ---------- 100.78.230.4:6379
- [root@qd01-stop-k8s-master001 redis]# kubectl -n op exec -it redis-cluster-shard0-0 -- redis-cli cluster info
- cluster_state:ok
- cluster_slots_assigned:16384
- cluster_slots_ok:16384
- cluster_slots_pfail:0
- cluster_slots_fail:0
- cluster_known_nodes:6
- cluster_size:3
- cluster_current_epoch:3
- cluster_my_epoch:1
- cluster_stats_messages_ping_sent:864
- cluster_stats_messages_pong_sent:882
- cluster_stats_messages_sent:1746
- cluster_stats_messages_ping_received:879
- cluster_stats_messages_pong_received:864
- cluster_stats_messages_meet_received:3
- cluster_stats_messages_received:1746
- [root@qd01-stop-k8s-master001 redis]# kubectl -n op exec -it redis-cluster-shard0-0 -- redis-cli cluster nodes
- 1895cb4b9c31b848666c61000e502f55a29a8255 100.64.147.155:6379@16379 master - 0 1615365162008 2 connected 5461-10922
- 30bdbf2ca37001774498a9b935afbc1cd2ce389c 100.126.252.203:6379@16379 slave 2c06092fafa99e0158e39e6237a04fed25be3550 0 1615365163000 1 connected
- 9b2cfbd5c1b417121d410141b6da9512ad29ce3c 100.78.230.3:6379@16379 slave e83446c368839c5fdccf5f70e3b1004eb67cb651 0 1615365163512 3 connected
- 2c06092fafa99e0158e39e6237a04fed25be3550 100.82.197.130:6379@16379 myself,master - 0 1615365162000 1 connected 0-5460
- 1379d2b20f26ab13d53068d276ec5d988b7a0273 100.64.122.197:6379@16379 slave 1895cb4b9c31b848666c61000e502f55a29a8255 0 1615365163000 2 connected
- e83446c368839c5fdccf5f70e3b1004eb67cb651 100.107.55.68:6379@16379 master - 0 1615365164014 3 connected 10923-16383
【編輯推薦】
責任編輯：姜華
來源：
今日頭條
