目錄

哨兵的作用和工作原理

服務(wù)狀態(tài)監(jiān)控

選舉新的 master

如何實(shí)現(xiàn)故障轉(zhuǎn)移

搭建哨兵集群

哨兵的作用和工作原理

Redis 提供了哨兵 (Sentinel) 機(jī)制來實(shí)現(xiàn)主從集群的自動(dòng)故障恢復(fù)。哨兵的結(jié)構(gòu)和作用如下

• 監(jiān)控：Sentinel 會(huì)不斷檢查你的 master 和 slave 是否按預(yù)期工作

• 自動(dòng)故障恢復(fù)：如果 master 故障， Sentinel 會(huì)將一個(gè) Slave 提成為 master ，當(dāng)故障實(shí)例恢復(fù)后也還是以新的 master 為主

• 通知：Sentinel 充當(dāng) Redis 客戶端的服務(wù)發(fā)現(xiàn)來源，當(dāng)集群發(fā)生故障轉(zhuǎn)移時(shí)，會(huì)將最新消息推送至 Redis 客戶端

服務(wù)狀態(tài)監(jiān)控

Sentinel 基于心跳機(jī)制檢測(cè)服務(wù)狀態(tài)，每隔 1 秒向集群的每個(gè)實(shí)例發(fā)送ping 命令

• 主管下線：如果某 Sentinel 節(jié)點(diǎn)發(fā)現(xiàn)某實(shí)例未在規(guī)定時(shí)間內(nèi)響應(yīng)，則認(rèn)為該實(shí)例主觀下線

• 客觀下線：若超過指定數(shù)量（quorum）的 sentinel 都認(rèn)為該實(shí)例主觀下線，則該實(shí)例客觀下線。quorum 值最好超過 Sentinel 實(shí)例數(shù)量的一半

選舉新的 master

一旦發(fā)現(xiàn) master 故障，sentinel 需要在 slave 中選擇一個(gè)作為新的 master ：

• 首先會(huì)判斷 slave 節(jié)點(diǎn)與 master 節(jié)點(diǎn)斷開的時(shí)間長(zhǎng)短，如果超出指定值 （down-after-milliseconds * 10）則會(huì)排除該 slave 節(jié)點(diǎn)

• 然后判斷 slave 節(jié)點(diǎn)的 slave-priority 值，越小優(yōu)先級(jí)越高，如果是 0 則永遠(yuǎn)不參與選舉

• 如果 slave-prority 一樣，則判斷 slave 節(jié)點(diǎn)的 offset 值，越大說明數(shù)據(jù)越新，優(yōu)先級(jí)越高

• 最后是判斷 slave 節(jié)點(diǎn)的運(yùn)行 id 大小，越小優(yōu)先級(jí)越高

如何實(shí)現(xiàn)故障轉(zhuǎn)移

當(dāng)選中了其中一個(gè) slave 為新的 master 后，故障的轉(zhuǎn)移的步驟如下

• sentinel 給備選的 slave 節(jié)點(diǎn)發(fā)送 slaveof no one 命令，讓該節(jié)點(diǎn)成為 master

• sentinel 給所有其他 slave 發(fā)送 slaveof 192.168.142.152 6379 命令，讓這些 slave 成為新的 master 的從節(jié)點(diǎn)，開始從新的 master 上同步數(shù)據(jù)

• 最后，sentinel 將故障節(jié)點(diǎn)標(biāo)記為 slave ，當(dāng)故障節(jié)點(diǎn)恢復(fù)后會(huì)自動(dòng)成為新的 master 的 slave 節(jié)點(diǎn)

搭建哨兵集群

IP	PORT	ROLE
192.168.142.157	6379	master
192.168.142.156	6379	slave
192.168.142.155	6379	slave02
192.168.142.157	26379	sentinel
192.168.142.156	26379	sentinel
192.168.142.155	26379	sentinel

我這里省事，只用了三臺(tái)服務(wù)

master , sentinel

docker-compose.yml

services:redis-master:image: hub.atomgit.com/amd64/redis:7.0.13restart: alwayscontainer_name: redis-masterprivileged: trueports:- '6379:6379'volumes:- redis-data:/opt/bitnami/redis/data- /root/redis.conf:/etc/redis.conf- /etc/localtime:/etc/localtime:rocommand:- /bin/sh- -c- redis-server /etc/redis.confredis-sentinel:image: hub.atomgit.com/amd64/redis:7.0.13restart: alwayscontainer_name: redis-sentinelprivileged: trueports:- '26379:26379'volumes:- /root/sentinel.conf:/etc/sentinel.conf- /etc/localtime:/etc/localtime:rocommand:- /bin/sh- -c- redis-server /etc/sentinel.conf --sentinel
volumes:redis-data:

redis.conf

daemonize no
port 6379
protected-mode no
bind 0.0.0.0
requirepass 123456

sentinel.conf

port 26379
protected-mode no
sentinel monitor mymaster 192.168.142.157 6379 2
sentinel auth-pass mymaster 123456
sentinel down-after-milliseconds mymaster 5000
sentinel parallel-syncs mymaster 1
sentinel failover-timeout mymaster 3000

slave , sentinel

docker-compose.yml

services:redis-slave:image: hub.atomgit.com/amd64/redis:7.0.13restart: alwayscontainer_name: redis-slaveprivileged: trueports:- '6379:6379'volumes:- redis-data:/opt/bitnami/redis/data- /root/redis.conf:/etc/redis.conf- /etc/localtime:/etc/localtime:rocommand:- /bin/sh- -c- redis-server /etc/redis.confredis-sentinel:image: hub.atomgit.com/amd64/redis:7.0.13restart: alwayscontainer_name: redis-sentinelprivileged: trueports:- '26379:26379'volumes:- /root/sentinel.conf:/etc/sentinel.conf- /etc/localtime:/etc/localtime:rocommand:- /bin/sh- -c- redis-server /etc/sentinel.conf --sentinel
volumes:redis-data:

redis.conf

daemonize no
port 6379
protected-mode no
masterauth 123456
requirepass 123456
slave-read-only yes
bind 0.0.0.0
slaveof 192.168.142.157 6379

sentinel.conf

port 26379
protected-mode no
sentinel monitor mymaster 192.168.142.157 6379 2
sentinel auth-pass mymaster 123456
sentinel down-after-milliseconds mymaster 5000
sentinel parallel-syncs mymaster 1
sentinel failover-timeout mymaster 3000

slave02 , sentinel

docker-compose.yml

services:redis-slave02:image: hub.atomgit.com/amd64/redis:7.0.13restart: alwayscontainer_name: redis-slave02privileged: trueports:- '6379:6379'volumes:- redis-data:/opt/bitnami/redis/data- /root/redis.conf:/etc/redis.conf- /etc/localtime:/etc/localtime:rocommand:- /bin/sh- -c- redis-server /etc/redis.confredis-sentinel:image: hub.atomgit.com/amd64/redis:7.0.13restart: alwayscontainer_name: redis-sentinelprivileged: trueports:- '26379:26379'volumes:- /root/sentinel.conf:/etc/sentinel.conf- /etc/localtime:/etc/localtime:rocommand:- /bin/sh- -c- redis-server /etc/sentinel.conf --sentinel
volumes:redis-data:

redis.conf

daemonize no
port 6379
protected-mode no
masterauth 123456
requirepass 123456
slave-read-only yes
bind 0.0.0.0
slaveof 192.168.142.157 6379

sentinel.conf

port 26379
protected-mode no
sentinel monitor mymaster 192.168.142.157 6379 2
sentinel auth-pass mymaster 123456
sentinel down-after-milliseconds mymaster 5000
sentinel parallel-syncs mymaster 1
sentinel failover-timeout mymaster 3000

哨兵就盯著 master 看，一旦 master g了就立刻預(yù)警，就可以開始啟動(dòng) docker 了

啟動(dòng) docker

在三臺(tái)主機(jī)上分別執(zhí)行

docker compose up -d

查看狀態(tài) Up 表示成功啟動(dòng)

root@master:~# docker ps -a
CONTAINER ID   IMAGE                                COMMAND                  CREATED          STATUS          PORTS                                                     NAMES
aa6466e6fa15   hub.atomgit.com/amd64/redis:7.0.13   "docker-entrypoint.s…"   27 minutes ago   Up 21 minutes   6379/tcp, 0.0.0.0:26379->26379/tcp, :::26379->26379/tcp   redis-sentinel
27a2f19d8040   hub.atomgit.com/amd64/redis:7.0.13   "docker-entrypoint.s…"   27 minutes ago   Up 21 minutes   0.0.0.0:6379->6379/tcp, :::6379->6379/tcp                 redis-master

 root@slave:~# docker ps -a
CONTAINER ID   IMAGE                                COMMAND                  CREATED             STATUS          PORTS                                                     NAMES
1d3df2c507f5   hub.atomgit.com/amd64/redis:7.0.13   "docker-entrypoint.s…"   25 minutes ago      Up 21 minutes   6379/tcp, 0.0.0.0:26379->26379/tcp, :::26379->26379/tcp   redis-sentinel
b9b981917f2d   hub.atomgit.com/amd64/redis:7.0.13   "docker-entrypoint.s…"   About an hour ago   Up 21 minutes   0.0.0.0:6379->6379/tcp, :::6379->6379/tcp                 redis-slave

root@slave02:~# docker ps -a
CONTAINER ID   IMAGE                                COMMAND                  CREATED             STATUS          PORTS                                                     NAMES
774fb813bbf2   hub.atomgit.com/amd64/redis:7.0.13   "docker-entrypoint.s…"   23 minutes ago      Up 20 minutes   6379/tcp, 0.0.0.0:26379->26379/tcp, :::26379->26379/tcp   redis-sentinel
02a276c8edc8   hub.atomgit.com/amd64/redis:7.0.13   "docker-entrypoint.s…"   About an hour ago   Up 20 minutes   0.0.0.0:6379->6379/tcp, :::6379->6379/tcp                 redis-slave02

啟動(dòng)成功就可以去查看 sentinel 的狀態(tài)了

root@master:~# docker exec redis-sentinel redis-cli -p 26379 -c info sentinel
# Sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_tilt_since_seconds:-1
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=mymaster,status=ok,address=192.168.142.157:6379,slaves=2,sentinels=1

root@slave:~# docker exec -it redis-sentinel redis-cli -p 26379 -c info sentinel
# Sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_tilt_since_seconds:-1
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=mymaster,status=ok,address=192.168.142.157:6379,slaves=0,sentinels=1

root@slave02:~# docker exec -it redis-sentinel redis-cli -p 26379 -c info sentinel
# Sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_tilt_since_seconds:-1
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=mymaster,status=ok,address=192.168.142.157:6379,slaves=0,sentinels=1

出現(xiàn)本機(jī) ip 和 ok 狀態(tài)就代表哨兵啟動(dòng)成功

測(cè)試

假設(shè) master 宕機(jī)

root@master:~# docker stop redis-master

查看 slave

root@slave:~# docker exec redis-slave redis-cli -a 123456 -c role
slave
192.168.142.157
6379
connect
-1
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.

查看 slave02

root@slave02:~# docker exec -it redis-slave02 redis-cli -a 123456 -c role
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
1) "master"
2) (integer) 19487
3) 1) 1) "192.168.142.156"2) "6379"3) "19347"

到此，哨兵搭建完成

補(bǔ)充

關(guān)于 sentinel.conf 講解

• sentinel monitor mymaster 192.168.142.157?6379 2 指定主節(jié)點(diǎn)信息

  • mymaster 主節(jié)點(diǎn)名稱，自定義

  • 192.168.142.157?6379 主節(jié)點(diǎn) IP 和端口

  • 2 選舉 master 時(shí)的 quorum 值

• sentinel down-after-milliseconds mymaster 5000 salve 與 master 斷開的超時(shí)時(shí)間

• sentinel failover-timeout mymaster 60000 故障恢復(fù)的超時(shí)時(shí)間

• sentinel auth-pass mymaster 123456 主節(jié)點(diǎn)密碼

• sentinel parallel-syncs mymaster 1
  這條指令告訴 Sentinel，對(duì)于名為 mymaster 的 Redis 主節(jié)點(diǎn)，在進(jìn)行故障轉(zhuǎn)移時(shí)，只允許一個(gè)從節(jié)點(diǎn)同時(shí)對(duì)新的主節(jié)點(diǎn)進(jìn)行數(shù)據(jù)同步。這意味著在故障轉(zhuǎn)移過程中，只有一個(gè)從節(jié)點(diǎn)會(huì)開始與新的主節(jié)點(diǎn)同步數(shù)據(jù)，其他從節(jié)點(diǎn)會(huì)等待，直到該從節(jié)點(diǎn)完成同步后才開始。

  設(shè)置 parallel-syncs 為 1 可以確保在故障轉(zhuǎn)移期間，只有一個(gè)從節(jié)點(diǎn)在任何給定時(shí)間與新的主節(jié)點(diǎn)進(jìn)行數(shù)據(jù)同步。這樣做的好處是可以減少對(duì)新主節(jié)點(diǎn)的負(fù)載，避免在故障轉(zhuǎn)移期間對(duì)新主節(jié)點(diǎn)造成過大的壓力，從而影響其性能。然而，這也意味著故障轉(zhuǎn)移過程可能會(huì)花費(fèi)更長(zhǎng)的時(shí)間，因?yàn)閺墓?jié)點(diǎn)需要一個(gè)接一個(gè)地進(jìn)行數(shù)據(jù)同步。