Setup Highly available Redis using Sentinel

About Redis

Redis is a well known and highly used in-memory database that persists data on disk. It is based on a key value data model, but it supports many kinds of values : Strings, Lists, Sets, Hashes, Bitmaps, Geo spatial data etc…

It can serve also as a message broker using the PUBSUB capabilities offered out of the box.

It is really straightforward to install it and get started, but in some scenarios we might want to have a highly available setup.

Redis Sentinel

Since the release of the version 2.8 in 2013, Redis marked Sentinel as a stable component to offer a high availability Redis cluster.

It supports the following capabilities :

• Monitoring : Sentinel operates in an autonomous mode to check if the master / replica nodes are operational and synchronized.
• Notification : It can be configured to notify any end in case something is going wrong via an API. This is quite useful to handle any incident the correct way.
• Automatic failover : In case of any problem within the master node, Sentinel promotes automatically a replica node to the master mode. Other replicas will point as well to the new master node.
• Configuration Maestro : Sentinel acts as the configuration source of truth, we can communicate with it in order to query information about the master node information, or any other available replica.

Under the hood of Sentinel

Sentinel operates as a totally independent node, you can see it as a monitoring tour. It keeps track of the health of all the available nodes in the cluster, and acts on its own when it comes to switching to the failover mode in order to promote another node to the master role.

Sentinel is a highly distributed system, and in some scenarios we might have multiple Sentinel nodes with a fact about the health of the master node. The decision to promote a replica is made after all the sentinels agree based on the quorum value.

Quorum Value

You can find a so similar concept in Biology and its called Quorum Sensing 

For Sentinel, the Quorum value represents the exact number of other Sentinels that need to agree about the health of the master node so an action can be taken. However, there is no further configuration around the Quorum.

One of the Sentinels takes the initiative to promote another node to be the master, but only if all the other Sentinels report back the same, and it got enough authorization to do so.

Redis Replication

Before proceeding, we need to be sure that we are in a Redis replicated environment, which means that we have a master node available to receive all write categorized commands, and a replica that we use to read from

You can run the following command in Redis CLI to check the replication information :

info replication

Sentinel Initial Setup

In this tutorial we will assume that our environment have the following IP addresses

Master Node: 10.132.0.10

Replica Node 1: 10.132.0.11

Replica Node 2: 10.132.0.12

We need to tell all the replica node who is the master node, let’s edit the following files :

vi /etc/redis.conf

vi /etc/redis-sentinel.conf

Be sure to have the following line containing the master node IP 

# Add this line to both configuration files
slaveof 10.132.0.10 6379

Now, tell Sentinel to monitor the available nodes :

vi /etc/redis-sentinel.conf

Add the following line in the master node :

sentinel monitor master-node 127.0.0.1 6379 2

You can replace `master-node` with any identifier of your choice, just be sure its the same on the other nodes.

Add the following line in the replica nodes 1 and 2 :

sentinel monitor master-node 10.132.0.10 6379 2

You might remark that we have number “2” at the end of the configuration line, which means 2 Sentinels needs to agree that the master node is faulty in order to promote a replica to the master role.

Password protected cluster :

If your Redis setup is protected by a password, be sure to add the following line in the `/etc/redis-sentinel.conf` file of all the nodes

sentinel auth-pass master-node REDIS_PASSWORD_HERE

Sentinel Configuration 

Note : In all of the cases when I interacted with Redis architectures, the timeout between nodes or failover checks never crossed 1-3 seconds maximum. Redis is considered as the highest performant in-memory cache. Rare cases when it happened, and it was due to some networking related issues.

Failover Timeout

This configuration flag tells Sentinel the number of milliseconds to wait until marking a master node +down. Sentinel will wait for the master health check reply for that amount of time. In the positive case of the timeout, then a +vote between the 2 sentinels begins to promote another node as a master.

# Failover after 10 seconds
sentinel failover-timeout master-node 10000

Down after Milliseconds

The time the node is not responding till its considered +down 

sentinel down-after-milliseconds master-node 3000

Starting Sentinel

Now, we are ready to start Sentinel to start monitoring our Redis stack.

In all the nodes execute the following command : 

systemctl restart redis

systemctl restart redis-sentinel

It might differ for each case, if you use any other system manager.

Now, all the nodes are going to be discovered, indexed by assigning a hash to each one to be identified by the monitor. You can check the nodes Sentinel configuration file to see the samples.

Using Redis CLI, you can run the following command to check the Sentinel status :

redis-cli -p 26379 info sentinel

Sentinel implements an API, that is accessible via CLI as well

Slaves discovery :

redis-cli -p 26379 sentinel slaves

Discovering the sentinels of a master node :

redis-cli -p 26379 sentinel sentinels master-node

Debugging the logs

Beside the Redis logs, now we can also debug the Sentinel logs :

tail -f /var/log/redis/redis.log

tail -f /var/log/redis/sentinel.log

Forcing a failover test

We can freeze the master node availability for some seconds using the debug sleep command, this will let us test if Sentinel will promote another slave as the master node.

Go ahead and run this in the master node using redis-cli :

redis-cli -p 6379 DEBUG sleep 120

Be sure the sleep time is higher than the timeout configuration flag.

Let’s now try to check who is the new master node :

redis-cli -p 26379 sentinel get-master-addr-by-name master-node

In my case I got 10.132.0.11, which is replica node 1, now it’s promoted to be the new master node.

Now, you have a highly available Redis cluster backed by the awesome capabilities of Sentinel.

Resources :

https://redis.io/topics/cluster-spec

https://redis.io/topics/sentinel