Redis is a well known and highly used in-memory database that persists data on disk. It is based on a key value data model, but it supports many kinds of values : Strings, Lists, Sets, Hashes, Bitmaps, Geo spatial data etc…
It can serve also as a message broker using the PUBSUB capabilities offered out of the box.
It is really straightforward to install it and get started, but in some scenarios we might want to have a highly available setup.
Since the release of the version 2.8 in 2013, Redis marked Sentinel as a stable component to offer a high availability Redis cluster.
It supports the following capabilities :
- Monitoring : Sentinel operates in an autonomous mode to check if the master / replica nodes are operational and synchronized.
- Notification : It can be configured to notify any end in case something is going wrong via an API. This is quite useful to handle any incident the correct way.
- Automatic failover : In case of any problem within the master node, Sentinel promotes automatically a replica node to the master mode. Other replicas will point as well to the new master node.
- Configuration Maestro : Sentinel acts as the configuration source of truth, we can communicate with it in order to query information about the master node information, or any other available replica.
Under the hood of Sentinel
Sentinel operates as a totally independent node, you can see it as a monitoring tour. It keeps track of the health of all the available nodes in the cluster, and acts on its own when it comes to switching to the failover mode in order to promote another node to the master role.
Sentinel is a highly distributed system, and in some scenarios we might have multiple Sentinel nodes with a fact about the health of the master node. The decision to promote a replica is made after all the sentinels agree based on the quorum value.
You can find a so similar concept in Biology and its called Quorum Sensing
For Sentinel, the Quorum value represents the exact number of other Sentinels that need to agree about the health of the master node so an action can be taken. However, there is no further configuration around the Quorum.
One of the Sentinels takes the initiative to promote another node to be the master, but only if all the other Sentinels report back the same, and it got enough authorization to do so.
Before proceeding, we need to be sure that we are in a Redis replicated environment, which means that we have a master node available to receive all write categorized commands, and a replica that we use to read from
You can run the following command in Redis CLI to check the replication information :
Sentinel Initial Setup
In this tutorial we will assume that our environment have the following IP addresses
Master Node: 10.132.0.10
Replica Node 1: 10.132.0.11
Replica Node 2: 10.132.0.12
We need to tell all the replica node who is the master node, let’s edit the following files :
Be sure to have the following line containing the master node IP
# Add this line to both configuration files
slaveof 10.132.0.10 6379
Now, tell Sentinel to monitor the available nodes :
Add the following line in the master node :
sentinel monitor master-node 127.0.0.1 6379 2
You can replace `master-node` with any identifier of your choice, just be sure its the same on the other nodes.
Add the following line in the replica nodes 1 and 2 :
sentinel monitor master-node 10.132.0.10 6379 2
You might remark that we have number “2” at the end of the configuration line, which means 2 Sentinels needs to agree that the master node is faulty in order to promote a replica to the master role.
Password protected cluster :
If your Redis setup is protected by a password, be sure to add the following line in the `/etc/redis-sentinel.conf` file of all the nodes
sentinel auth-pass master-node REDIS_PASSWORD_HERE
Note : In all of the cases when I interacted with Redis architectures, the timeout between nodes or failover checks never crossed 1-3 seconds maximum. Redis is considered as the highest performant in-memory cache. Rare cases when it happened, and it was due to some networking related issues.
This configuration flag tells Sentinel the number of milliseconds to wait until marking a master node +down. Sentinel will wait for the master health check reply for that amount of time. In the positive case of the timeout, then a +vote between the 2 sentinels begins to promote another node as a master.
# Failover after 10 seconds
sentinel failover-timeout master-node 10000
Down after Milliseconds
The time the node is not responding till its considered +down
sentinel down-after-milliseconds master-node 3000
Now, we are ready to start Sentinel to start monitoring our Redis stack.
In all the nodes execute the following command :
systemctl restart redis
systemctl restart redis-sentinel
It might differ for each case, if you use any other system manager.
Now, all the nodes are going to be discovered, indexed by assigning a hash to each one to be identified by the monitor. You can check the nodes Sentinel configuration file to see the samples.
Using Redis CLI, you can run the following command to check the Sentinel status :
redis-cli -p 26379 info sentinel
Sentinel implements an API, that is accessible via CLI as well
Slaves discovery :
redis-cli -p 26379 sentinel slaves
Discovering the sentinels of a master node :
redis-cli -p 26379 sentinel sentinels master-node
Debugging the logs
Beside the Redis logs, now we can also debug the Sentinel logs :
tail -f /var/log/redis/redis.log
tail -f /var/log/redis/sentinel.log
Forcing a failover test
We can freeze the master node availability for some seconds using the debug sleep command, this will let us test if Sentinel will promote another slave as the master node.
Go ahead and run this in the master node using redis-cli :
redis-cli -p 6379 DEBUG sleep 120
Be sure the sleep time is higher than the timeout configuration flag.
Let’s now try to check who is the new master node :
redis-cli -p 26379 sentinel get-master-addr-by-name master-node
In my case I got 10.132.0.11, which is replica node 1, now it’s promoted to be the new master node.
Now, you have a highly available Redis cluster backed by the awesome capabilities of Sentinel.