Kubernetes（k8s）搭建部署redis cluster集群

✍️Auth:运维笔记       Date:2025/04/19       Cat:Database、Linux服务器       👁️:57 次浏览

  • 如果 Redis 是关键业务组件，并且性能要求极高（低延迟、百万 QPS），使用官方 Redis Cluster 直接部署在裸机或 VM 上更合适。
  • 如果 Redis 主要是缓存层，且你的基础设施是 Kubernetes，使用 StatefulSet + Redis Operator 更方便管理和扩展。

裸机主机搭建参考文章：主机模式搭建redis cluster集群

1，安装nfs 服务

推荐在生产中使用 NFS 或其他持久化存储服务。这里使用NFS存储服务。

生产环境中通常会部署独立的 NFS 服务（比如用 NAS、GlusterFS、CephFS），而不是依赖容器内运行的 NFS Server。所以单独部署在主机上，直接跑在宿主机上意味着 更少的网络转发和 NAT，对于 Redis 这种 IO 敏感的服务，延迟更低、吞吐更高。

安装 nfs ,示例IP：172.16.81.129

sudo apt update
sudo apt install -y nfs-kernel-server
​
# 创建共享目录
sudo mkdir -p /srv/nfs/redis-data
sudo chown nobody:nogroup /srv/nfs/redis-data
​
# 编辑导出目录
echo "/srv/nfs/redis-data *(rw,sync,no_subtree_check,no_root_squash)" | sudo tee -a /etc/exports
​
# 重启服务
sudo exportfs -rav
sudo systemctl restart nfs-kernel-server

2，安装NFS CSI 驱动实现自动创建 PV（动态供给）

NFS CSI Driver 是用来动态自动创建 PV（动态供给）的。Redis-0 的 PVC（data-redis-0）没有绑定成功的 PV，整个 Pod就会无法调度。会报错pod has unbound immediate PersistentVolumeClaims。

— 手动创建PV —

不想安装插件就手动创建6个PV。

#0-5依次对应pv位置
redis-0  >>  /srv/nfs/redis-data/redis-0   
....
redis-5  >>  /srv/nfs/redis-data/redis-5  

在nfs主机创建6个目录

sudo mkdir -p /srv/nfs/redis-data/redis-{0..5}
sudo chown -R nobody:nogroup /srv/nfs/redis-data
sudo exportfs -rav

主节点创建 pv-redis.yaml 文件。

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-redis-0
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: nfs-host
  nfs:
    server: 172.16.81.129
    path: /srv/nfs/redis-data/redis-0

以上文件复制 6 次改掉 pv-redis-X 和 path 路径即可。

最后执行部署6个文件即可，pv就创建完成了。

kubectl apply -f pv-redis-0.yaml
....
kubectl apply -f pv-redis-5.yaml

— 安装NFS CSI自动创建PV —

嫌手动麻烦，可安装官方 NFS CSI Driver，然后 Redis 的 PVC 只要写 storageClassName: nfs-csi，就能自动在 NFS 主机上动态创建目录并绑定。

官方地址：https://github.com/kubernetes-csi/csi-driver-nfs/tree/master

#在线安装（目前最新版本v4.11.0）：
curl -skSL https://raw.githubusercontent.com/kubernetes-csi/csi-driver-nfs/v4.11.0/deploy/install-driver.sh | bash -s v4.11.0 --

#下载本地安装
git clone https://github.com/kubernetes-csi/csi-driver-nfs.git
cd csi-driver-nfs
./deploy/install-driver.sh v4.11.0 local

安装完成后可查看服务。

root@master:/opt/redis-k8s# kubectl -n kube-system get pod -o wide -l app=csi-nfs-controller
root@master:/opt/redis-k8s# kubectl -n kube-system get pod -o wide -l app=csi-nfs-node
​
NAME                 READY   STATUS    RESTARTS      AGE   IP              NODE     NOMINATED NODE   READINESS GATES
csi-nfs-controller... 5/5    Running   2 (2m ago)    3m   172.16.81.129   node01   <none>           <none>
csi-nfs-node-hcm4b   3/3     Running   1 (2m ago)    3m   172.16.81.129   node01   <none>           <none>
csi-nfs-node-v4r5b   3/3     Running   1 (2m ago)    3m   172.16.81.128   master   <none>           <none>

最后部署redis完成后不需要也可以卸载掉。

#在线卸载
curl -skSL https://raw.githubusercontent.com/kubernetes-csi/csi-driver-nfs/v4.11.0/deploy/uninstall-driver.sh | bash -s v4.11.0 --

#本地卸载
git clone https://github.com/kubernetes-csi/csi-driver-nfs.git
cd csi-driver-nfs
git checkout v4.11.0
./deploy/uninstall-driver.sh v4.11.0 local

3，redis集群yaml文件

文件目录结构如下：

root@master:/opt/redis-k8s# 
.
├── 1-namespace.yaml
├── 2-redis-cluster.yaml
├── 3-init-cluster-job.yaml
└── storageclass-nfs.yaml
文件名说明
1-namespace.yaml创建 redis-cluster 命名空间（可以复用）
2-redis-cluster.yamlConfigMap + Service（2个）+ StatefulSet（6节点，挂载 NFS）
3-init-cluster-job.yaml创建 Redis 集群初始化 Job（自动主从+分槽）
storageclass-nfs.yaml定义使用 NFS 的 StorageClass（绑定到NFS）

第二个文件2-redis-cluster.yaml，如果想更清晰，分开管理，可以分离单独的3个文件部署。

配置文件ConfigMap一个，Service一个，主体StatefulSet1个即可，这里统一管理，就不分了。

— storageclass-nfs.yaml —

StorageClass 指向 NFS 主机。

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nfs-csi
provisioner: nfs.csi.k8s.io
parameters:
  server: 172.16.81.129     # 你的 NFS 服务地址
  share: /srv/nfs/redis-data # 你导出的 NFS 路径
reclaimPolicy: Retain
volumeBindingMode: Immediate

这里用的是主路径 /srv/nfs/redis-data，CSI Driver 会自动为每个 PVC 创建子目录，比如创建完成后nfs目录：

ubuntu@node01:~$ ls /srv/nfs/redis-data/
pvc-1a9349a7-afca-49f6-b073-e45dabcc5365  pvc-57d7a764-2101-4f8b-8e7c-91512e028ad9  pvc-88d9b066-7ed7-4a77-ba55-cabf8d7a47d1
pvc-5777dc53-4815-4047-b3bd-dd49dbe6d3bf  pvc-5992e08d-88c4-4b9e-a4c5-51e373c88766  pvc-b55fe3d4-0b44-42b3-a0f8-bcef0528f25d

— 1-namespace.yaml —

apiVersion: v1
kind: Namespace
metadata:
  name: redis-cluster

须独立，作用范围全局

— 2-redis-cluster.yaml —

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: redis-config
  namespace: redis-cluster
data:
  redis.conf: |
    port 6379
    requirepass passwd123
    masterauth passwd123
    cluster-enabled yes
    cluster-config-file nodes.conf
    cluster-node-timeout 5000
    appendonly yes
    dir /data
---
apiVersion: v1
kind: Service
metadata:
  name: redis
  namespace: redis-cluster
spec:
  clusterIP: None
  selector:
    app: redis
  ports:
    - port: 6379
---
apiVersion: v1
kind: Service
metadata:
  name: redis-access
  namespace: redis-cluster
spec:
  type: ClusterIP
  selector:
    app: redis
  ports:
    - port: 6379
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: redis
  namespace: redis-cluster
spec:
  serviceName: redis
  replicas: 6
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
    spec:
      containers:
        - name: redis
          image: redis:7.2
          command: [ "redis-server", "/usr/local/etc/redis/redis.conf" ]
          ports:
            - containerPort: 6379
          volumeMounts:
            - name: data
              mountPath: /data
            - name: config
              mountPath: /usr/local/etc/redis/redis.conf
              subPath: redis.conf
      volumes:
        - name: config
          configMap:
            name: redis-config
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: [ "ReadWriteOnce" ]
        storageClassName: nfs-csi
        resources:
          requests:
            storage: 1Gi

配置文件里面两个服务：

  • 第一个是无头服务，允许 StatefulSet 使用 DNS 发现（如 redis-0.redis.redis-cluster.svc.cluster.local），仅集群内部服务访问使用。
  • 第二个是普通服务，用于在初始化集群时作为一个公共连接入口。可按需更改Ingress 或 NodePort暴露端口给外部访问。

可以保留两个，职责不同, 可选优化：如果你通过 kubectl exec 初始化，不需要 redis-access service。

— 3-init-cluster-job.yaml —

初始化集群

apiVersion: batch/v1
kind: Job
metadata:
  name: redis-init-cluster
  namespace: redis-cluster
spec:
  template:
    spec:
      containers:
        - name: redis-init
          image: redis:7.2
          command:
            - sh
            - -c
            - |
              echo "Waiting for Redis pods to be ready...";
              sleep 10;
              redis-cli --cluster create \
                redis-0.redis:6379 \
                redis-1.redis:6379 \
                redis-2.redis:6379 \
                redis-3.redis:6379 \
                redis-4.redis:6379 \
                redis-5.redis:6379 \
                --cluster-replicas 1 -a passwd123 --cluster-yes
      restartPolicy: OnFailure

4，部署

— 部署顺序 —

kubectl apply -f storageclass-nfs.yaml
kubectl apply -f 1-namespace.yaml
kubectl apply -f 2-redis-cluster.yaml

#等待所有在 redis-cluster 命名空间中，带有 app=redis 标签的 Pod，进入 Ready 状态，最多等 300 秒。
kubectl wait --for=condition=Ready pod -l app=redis -n redis-cluster --timeout=300s

#node都ready后执行初始化集群
kubectl apply -f 3-init-cluster-job.yaml

— 集群查看验证 —

查看集群

root@master:/opt/redis-k8s# kubectl get pods -n redis-cluster -o wide
NAME                       READY   STATUS      RESTARTS   AGE     IP               NODE     NOMINATED NODE   READINESS GATES
redis-0                    1/1     Running     0          15m     10.244.196.135   node01   <none>           <none>
redis-1                    1/1     Running     0          4m5s    10.244.219.76    master   <none>           <none>
redis-2                    1/1     Running     0          4m2s    10.244.196.136   node01   <none>           <none>
redis-3                    1/1     Running     0          3m59s   10.244.219.77    master   <none>           <none>
redis-4                    1/1     Running     0          3m57s   10.244.196.137   node01   <none>           <none>
redis-5                    1/1     Running     0          3m55s   10.244.196.138   node01   <none>           <none>
redis-init-cluster-fgdsn   0/1     Completed   0          27s     10.244.196.139   node01   <none>           <none>

查看初始化日志

root@master:/opt/redis-k8s# kubectl logs -n redis-cluster pod/redis-init-cluster-fgdsn
Waiting for Redis pods to be ready...
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica redis-4.redis:6379 to redis-0.redis:6379
Adding replica redis-5.redis:6379 to redis-1.redis:6379
Adding replica redis-3.redis:6379 to redis-2.redis:6379
M: 9aecccf8d6857a6d67a0f5b52ce85dbf9c667b78 redis-0.redis:6379
   slots:[0-5460] (5461 slots) master
M: 281228e13747ebc41577979243d5166846ea2c81 redis-1.redis:6379
   slots:[5461-10922] (5462 slots) master
M: 970d60926019635d605c8d999eccaf69343099cd redis-2.redis:6379
   slots:[10923-16383] (5461 slots) master
S: c0fbc76dd0002db204842b8cb6f434f97dcb832e redis-3.redis:6379
   replicates 970d60926019635d605c8d999eccaf69343099cd
S: 202e3cf341c722817cac1248a2a413cd1a2d72fb redis-4.redis:6379
   replicates 9aecccf8d6857a6d67a0f5b52ce85dbf9c667b78
S: 8e0e18c60ae05eefb1fb0b722d0dc05e1774bbb9 redis-5.redis:6379
   replicates 281228e13747ebc41577979243d5166846ea2c81
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join

>>> Performing Cluster Check (using node redis-0.redis:6379)
M: 9aecccf8d6857a6d67a0f5b52ce85dbf9c667b78 redis-0.redis:6379
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
S: 202e3cf341c722817cac1248a2a413cd1a2d72fb 10.244.196.137:6379
   slots: (0 slots) slave
   replicates 9aecccf8d6857a6d67a0f5b52ce85dbf9c667b78
S: 8e0e18c60ae05eefb1fb0b722d0dc05e1774bbb9 10.244.196.138:6379
   slots: (0 slots) slave
   replicates 281228e13747ebc41577979243d5166846ea2c81
M: 970d60926019635d605c8d999eccaf69343099cd 10.244.196.136:6379
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
M: 281228e13747ebc41577979243d5166846ea2c81 10.244.219.76:6379
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
S: c0fbc76dd0002db204842b8cb6f434f97dcb832e 10.244.219.77:6379
   slots: (0 slots) slave
   replicates 970d60926019635d605c8d999eccaf69343099cd
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

初始化也没问题，集群就部署完了。

查看无头服务dns解析也没问题。

root@master:/opt/redis-k8s# kubectl get svc -n kube-system
NAME       TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
kube-dns   ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   13d

root@master:/opt/redis-k8s# nslookup redis.redis-cluster.svc.cluster.local 10.96.0.10
Server:         10.96.0.10
Address:        10.96.0.10#53

Name:   redis.redis-cluster.svc.cluster.local
Address: 10.244.196.135
Name:   redis.redis-cluster.svc.cluster.local
Address: 10.244.196.138
Name:   redis.redis-cluster.svc.cluster.local
Address: 10.244.219.77
Name:   redis.redis-cluster.svc.cluster.local
Address: 10.244.196.137
Name:   redis.redis-cluster.svc.cluster.local
Address: 10.244.219.76
Name:   redis.redis-cluster.svc.cluster.local
Address: 10.244.196.136

命令测试

root@master: kubectl exec -it redis-0 -n redis-cluster -- redis-cli -a passwd123 -c cluster nodes
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
281228e13747ebc41577979243d5166846ea2c81 10.244.219.76:6379@16379 myself,master - 0 1745036774000 2 connected 5461-10922
202e3cf341c722817cac1248a2a413cd1a2d72fb 10.244.196.137:6379@16379 slave 9aecccf8d6857a6d67a0f5b52ce85dbf9c667b78 0 1745036774000 1 connected
9aecccf8d6857a6d67a0f5b52ce85dbf9c667b78 10.244.196.135:6379@16379 master - 0 1745036775128 1 connected 0-5460
c0fbc76dd0002db204842b8cb6f434f97dcb832e 10.244.219.77:6379@16379 slave 970d60926019635d605c8d999eccaf69343099cd 0 1745036774120 3 connected
8e0e18c60ae05eefb1fb0b722d0dc05e1774bbb9 10.244.196.138:6379@16379 slave 281228e13747ebc41577979243d5166846ea2c81 0 1745036775128 2 connected
970d60926019635d605c8d999eccaf69343099cd 10.244.196.136:6379@16379 master - 0 1745036775027 3 connected 10923-16383

root@master: kubectl exec -it redis-0 -n redis-cluster -- redis-cli -a passwd123 -c cluster info
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:2
cluster_stats_messages_ping_sent:107940
cluster_stats_messages_pong_sent:110904
cluster_stats_messages_meet_sent:1
cluster_stats_messages_sent:218845
cluster_stats_messages_ping_received:110904
cluster_stats_messages_pong_received:107941
cluster_stats_messages_received:218845
total_cluster_links_buffer_limit_exceeded:0


root@master: kubectl exec -it redis-1 -n redis-cluster -- redis-cli -a passwd123 -c 
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
127.0.0.1:6379> set redis hello
-> Redirected to slot [1151] located at 10.244.196.135:6379
OK
10.244.196.135:6379> exit

root@master: kubectl exec -it redis-2 -n redis-cluster -- redis-cli -a passwd123 -c 
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
127.0.0.1:6379> get redis
-> Redirected to slot [1151] located at 10.244.196.135:6379
"hello"
10.244.196.135:6379>
打赏作者

《Kubernetes（k8s）搭建部署redis cluster集群》有1个想法

  1. Pingback引用通告： 主机模式搭建redis cluster集群 - 运维笔记（ywbj.cc）

发表评论