How to achieve fault tolerance in Redis network with Redis cluster and sentinels.
In this tutorial I am going to cover
-
Introduction of Redis
-
Create a spring boot application with redis cache using docker
-
Look at different ways to build fault tolerent redis network
- Sharding with a redis cluster
- Replication with redis cluster
- Redis master/slave network with sentinel (no sharding)
Let’s begin with an introduction to Redis for audience not familiar with Redis.
Introduction to Redis
Redis is an open source (BSD licensed), in-memory data structure store, used as a database, cache and message broker.
Redis has built-in replication, LRU eviction, transactions and different levels of on-disk persistence, and provides high availability via Redis Sentinel and automatic partitioning with Redis Cluster.
In order to achieve its outstanding performance, Redis works in-memory.
Some key Redis features to note before we dig into coding :
-
Maximum memory
By default redis have no memory limits on 64-bit systems and 3 GB on 32-bit systems. Large memory can contain more data and increase hit ratio, one of the most important metrics but at a certain limit of memory the hit rate will be at the same level.
-
Eviction algorithms
When cache size reaches the memory limit, old data is removed to make place for new one. Redis offer Last Recently Used and Least Frequently Used eviction algorithms.
- RDB point-in-time snapshots after specific interval of time or amount of writes
- AOF creates persistence logs with every write operation.
-
Durability
For some reasons you may want to persist your cache. After startup, cache is initially empty, it will be useful to fulfill it with snapshot data in case of recovery after outage. Redis support different ways to achieve the persistance.
Develope spring boot + redis + docker app
By the end of this article I will make several updates to github repo for this demo. Have a look at releases if you would like to look at code at different points in this article.
https://github.com/avinash10584/spring-boot-redis-cluster.git
Let’s build our redis app,
Step 1: Create a Simple Spring Boot TODO List app
As a first step to build the demo app, lets create a simple TODO list app. We will not be creating it for multiple users to keep it simple.
For now we simply store one simple TODO list in our app as cache. We are not using any database for now.
Step 2: Download and start Redis Docker image
Download official Redis image from docker hub typing
docker pull redis
After this command new image should be present in your local repository (type docker images to check it).
The project in github is configured to use both standalone and cluster mode.
Let’s first use redis in standalone mode. Simply start the redis image we pulled from dockerhub
docker run --rm -p 4025:6379 -d --name redis-1 redis redis-server
Step 3: Integrating Redis to the spring boot application
We will be using spring boot cache to talk to redis. Spring comes with several annotations that you can add to work Redis cache.
Add @EnableCaching in your Application config to enable these annotations,
Now let’s build our docker app to make sure spring boot app can talk to Redis,
Dockerfile for the project is located at the girhub repo, I am focusing on Redis so I will avoid details of the Dockerfile for the spring boot app.
If you want to learn more about the Dockerfile in use for the project and understand how to avoid docker build time and use caching then look at the below article.
https://codingfullstack.com/java/spring-boot/docker-spring-boot-guide/
DOCKER_BUILDKIT=1 docker build -t learnings/spring-boot-redis-cluster .
Once our image is built, the next step is to run the image,
docker run --rm -p 4024:4024 --name spring-boot-redis learnings/spring-boot-redis-cluster
If you run the application and visit http://localhost:4024/app/ you would get the error .ConnectTimeoutException: connection timed out.
This is simply because we want our two docker images to talk to each other and by default they sit on their own network and are isolated from each other.
We have two options at this point,
- Use Docker network
- Create a docker-compose that builds a default network
I will create the docker network for this demo, let me know in comments if you would like a docker-compose file added to project.
docker network create spring-redis-network
Now let’s connect our redis and spring boot images to this network,
docker network connect spring-redis-network redis-1
We can now look for ipaddress of our redis instance and update in application.yml
You can avoid this look up if we use docker-compose as it can bind services without giving specifics of ip addresses
docker inspect spring-redis-network
In my case the ip address was 172.18.0.2
"Name": "redis-1",
"EndpointID": "88b100f3569bb4ed68ac8cbf84f4b5a20493e11c5e7336a052bbbd25bb5f4205",
"MacAddress": "02:42:ac:12:00:02",
"IPv4Address": "172.18.0.2/16",
"IPv6Address": ""
We can now update this in application.yml
redis:
host: 172.18.0.2
port: 6379
Note that we use the port 6379 as we are in docker network, exposed port to the host is 4025
Now let’s build our image for app one more time and connect to network we created,
DOCKER_BUILDKIT=1 docker build -t learnings/spring-boot-redis-cluster .
We can connect to a network by passing --net when running our docker image
docker run --rm --net spring-redis-network -p 4024:4024 --name spring-boot-redis learnings/spring-boot-redis-cluster
You should see the todo list items at http://localhost:4024/app/
We can verify cache is created in our docker redis image
docker exec -it redis-1 redis-cli --scan
Step 4: Modify our app to Use Spring cache annotations
There are 5 basic annotations that you would normally use with Spring Cache
- @CachePut - This is used to update cache
- @Cacheable - To return cached response for a method
- @CacheEvict - To remove the cache entry no longer needed, thing delete for an entity
- @Caching - Java does not allow you to use same annotation type twice on a method or class, so If you want to say @CacheEvict two different caches on same method then the @Cacheable can be used to aggregate other cache annotations.
I have added these to our ToDoListController and it looks like below,
Let’s run our app again and check redis stats,
DOCKER_BUILDKIT=1 docker build -t learnings/spring-boot-redis-cluster .
docker run --rm --net spring-redis-network -p 4024:4024 --name spring-boot-redis learnings/spring-boot-redis-cluster
docker exec -it redis-1 redis-cli info stats
If you would like to check the code upto this point then have a look at the tag,
https://github.com/avinash10584/spring-boot-redis-cluster/releases/tag/REDIS-CACHE
Sharding with Redis Clusters
We have built a basic spring boot application with redis cache.
But what if we are dealing with large data that can’t be contained in one node. Redis supports clusters to shard your data across multiple nodes.
The entire keyspace in Redis Clusters is divided in 16384 slots (called hash slots) and these slots are assigned to multiple Redis nodes. A given key is mapped to one of these slots, and the hash slot for a key is computed as:
HASH_SLOT = CRC16(key) mod 16384
In most cases you don’t need to know these internals as Redis will take care of push/pull of data from right cluster.
Let’s stop our redis image and build a cluster,
docker stop redis-1
docker stop spring-boot-redis
Let’s spin two more redis nodes to build a cluster, we also pass a redis config file located in project/redis-conf.
Redis requires a minimum of 3 clusters to work.
Redis images have redis cluster support disabled by default so we need to add
a config file and pass that to our redis docker images, I have added this in project root under /redis-conf
Let’s start the shards,
docker run --rm --net spring-redis-network -v /mnt/c/Development/github/spring-boot-redis-cluster/redis-conf:/redis_config -p 4025:6379 -d --name redis-1 redis redis-server /redis_config/node1.conf
docker run --rm --net spring-redis-network -v /mnt/c/Development/github/spring-boot-redis-cluster/redis-conf:/redis_config -p 4026:6379 -d --name redis-2 redis redis-server /redis_config/node2.conf
docker run --rm --net spring-redis-network -v /mnt/c/Development/github/spring-boot-redis-cluster/redis-conf:/redis_config -p 4027:6379 -d --name redis-3 redis redis-server /redis_config/node3.conf
Now the redis images have started in cluster mode but we still need to create a cluster to bind them together, we can do a master slave configuration but for now we just need data sharding and we are creating cluster without failover.
We will look at failover in next section.
Run below to create a cluster
docker exec -it redis-1 redis-cli --cluster create 172.18.0.2:6379 172.18.0.3:6379 172.18.0.4:6379
you should see something like below in your output.
We can inspect our docker network to get new ip address for the redis nodes
docker inspect spring-redis-network
Our application-cluster.yml looks like below,
Let’s stop our docker spring boot app and relauch with cluster configuration,
docker stop spring-boot-redis
DOCKER_BUILDKIT=1 docker build -t learnings/spring-boot-redis-cluster .
docker run --rm --net spring-redis-network -e "SPRING_PROFILES_ACTIVE=cluster" -p 4024:4024 --name spring-boot-redis learnings/spring-boot-redis-cluster
You can refresh the app in browser to verify the application is working fine.
We can also verify the cluster configuration in any node
docker exec -it redis-2 redis-cli cluster nodes
Sharding allows our data to be distributed in multiple nodes for large datasets and also reduces the lookup by hashing.
Our cluster is missing two key safety checks of distributed system, failover handling and replication.
Replication with redis cluster
Redis cluster allows us to achieve failover handling and replication.
We’re going to setup our nodes in a master/slave configuration where we will have 1 master and 2 slave nodes.
This way, if we lose one node, the cluster will still be able to elect a new master. In this setup, writes will have to go through the master as slaves are read-only.
The upside to this is that if the master disappears, its entire state has already been replicated to the slave nodes, meaning when one is elected as master, it can begin to accept writes immediately.
Do we need sentinels ?
Sentinals are separate Redis instances that run alongside the redis node to decide their part in the cluster and also change master/slave as necessary in case of failover.
You don’t need Sentinel when using Redis cluster.
Redis Cluster perform automatic failover if any problem happens to any master instance.
First we should convert our cluster to master/slave.
Let’s add 3 more nodes that will work as slave of our 3 master nodes.
Let’s stop our nodes and start again,
docker stop redis-1
docker stop redis-2
docker stop redis-3
# Start redis nodes
docker run --rm --net spring-redis-network -v /mnt/c/Development/github/spring-boot-redis-cluster/redis-conf:/redis_config -p 4025:6379 -d --name redis-1 redis redis-server /redis_config/node1.conf
docker run --rm --net spring-redis-network -v /mnt/c/Development/github/spring-boot-redis-cluster/redis-conf:/redis_config -p 4026:6379 -d --name redis-2 redis redis-server /redis_config/node2.conf
docker run --rm --net spring-redis-network -v /mnt/c/Development/github/spring-boot-redis-cluster/redis-conf:/redis_config -p 4027:6379 -d --name redis-3 redis redis-server /redis_config/node3.conf
# Start slaves
docker run --rm --net spring-redis-network -v /mnt/c/Development/github/spring-boot-redis-cluster/redis-conf:/redis_config -p 5025:6379 -d --name redis-1-slave redis redis-server /redis_config/node1-slave.conf
docker run --rm --net spring-redis-network -v /mnt/c/Development/github/spring-boot-redis-cluster/redis-conf:/redis_config -p 5026:6379 -d --name redis-2-slave redis redis-server /redis_config/node2-slave.conf
docker run --rm --net spring-redis-network -v /mnt/c/Development/github/spring-boot-redis-cluster/redis-conf:/redis_config -p 5027:6379 -d --name redis-3-slave redis redis-server /redis_config/node3-slave.conf
Let’s inspect our docker network as we would need ip addresses to create the cluster
docker inspect spring-redis-network
docker exec -it redis-1 redis-cli --cluster create 172.18.0.2:6379 172.18.0.3:6379 172.18.0.4:6379 172.18.0.6:6379 172.18.0.7:6379 172.18.0.8:6379 --cluster-replicas 1
If you see the below error then stop you running nodes as they have in memory data and they need to be empty when creating cluster
[ERR] Node 172.18.0.3:6379 is not empty. Either the node already knows other nodes (check with CLUSTER NODES) or contains some key in database 0.
If all goes well you should see below output,
We can verify our cluster with,
docker exec -it redis-2 redis-cli cluster nodes
Our final application-cluster.yml looks like below,
Let’s run our app again and check redis stats,
DOCKER_BUILDKIT=1 docker build -t learnings/spring-boot-redis-cluster .
docker run --rm --net spring-redis-network -p 4024:4024 --name spring-boot-redis learnings/spring-boot-redis-cluster
docker exec -it redis-1 redis-cli info stats
Let’s stop one of our server docker stop redis-2
If you run docker exec -it redis-1 redis-cli cluster nodes again you would notice that slave is promoted as master.
In case of a slave failure redis automatically promotes a slave to master.
Redis master/slave network with sentinel (no sharding)
In scenarios where you don’t need sharding and and just one node is enough for your in memory needs then you can avoid creating clusters and build master slave by specifying the slaveof in the slave configurations for redis node.
If we are not using redis clusters then we need sentinels to achieve failovers.
The sentinels are different concept than redis cluster. If the master dies then sentinels talk with each other to decide new master.
Since sentinel configuration is very different from clusters, I have put this config in
/redis-conf-sentinel
Let’s add our sentinel servers so we have automatic failover,
Sentinels only need to look at master nodes to decide on failover.
sentinel monitor redis-cluster 172.18.0.2 6379 2
We can use below config to decide on how long before a cluster node is considered down.
sentinel down-after-milliseconds redis-cluster 5000
We can add below to allow timeout for current replication writes to complete before a failover kick-off,
sentinel failover-timeout redis-cluster 10000
Let’s stop all our docker containers
docker stop $(docker ps -a -q)
docker run --rm --net spring-redis-network -v /mnt/c/Development/github/spring-boot-redis-cluster/redis-conf-sentinel:/redis_config -p 4025:6379 -d --name redis-1 redis redis-server /redis_config/node1.conf
docker run --rm --net spring-redis-network -v /mnt/c/Development/github/spring-boot-redis-cluster/redis-conf-sentinel:/redis_config -p 5025:6379 -d --name redis-1-slave redis redis-server /redis_config/node1-slave-1.conf
docker run --rm --net spring-redis-network -v /mnt/c/Development/github/spring-boot-redis-cluster/redis-conf-sentinel:/redis_config -p 5026:6379 -d --name redis-2-slave redis redis-server /redis_config/node1-slave-2.conf
docker run --rm --net spring-redis-network -v /mnt/c/Development/github/spring-boot-redis-cluster/redis-conf-sentinel:/redis_config -p 6025:6379 -d --name sentinel-1 redis redis-server /redis_config/sentinel1.conf --sentinel
docker run --rm --net spring-redis-network -v /mnt/c/Development/github/spring-boot-redis-cluster/redis-conf-sentinel:/redis_config -p 6026:6379 -d --name sentinel-2 redis redis-server /redis_config/sentinel2.conf --sentinel
docker run --rm --net spring-redis-network -v /mnt/c/Development/github/spring-boot-redis-cluster/redis-conf-sentinel:/redis_config -p 6027:6379 -d --name sentinel-3 redis redis-server /redis_config/sentinel3.conf --sentinel
docker logs sentinel-2
We would need to update our spring boot app to also use sentinel configuration,
let’s add a separate application-sentinel.yml and start application
docker stop spring-boot-redis
DOCKER_BUILDKIT=1 docker build -t learnings/spring-boot-redis-cluster .
docker run --rm --net spring-redis-network -e "SPRING_PROFILES_ACTIVE=sentinel" -p 4024:4024 --name spring-boot-redis learnings/spring-boot-redis-cluster
We can stop our master instance to verify sentinels are working,
docker stop redis-1
You should see logs like below in your sentinel docker logs sentinel-1
1:X 18 Jun 2020 21:25:06.046 # +sdown master redis-cluster 172.18.0.2 6379
1:X 18 Jun 2020 21:25:07.679 # +new-epoch 1
1:X 18 Jun 2020 21:25:07.891 # +vote-for-leader bfebc5c7d07121c78633024dcbc89a14bf1e4563 1
1:X 18 Jun 2020 21:25:07.948 # +odown master redis-cluster 172.18.0.2 6379 #quorum 3/2
1:X 18 Jun 2020 21:25:07.948 # Next failover delay: I will not start a failover before Thu Jun 18 21:25:28 2020
1:X 18 Jun 2020 21:25:08.580 # +config-update-from sentinel bfebc5c7d07121c78633024dcbc89a14bf1e4563 172.18.0.7 6379 @ redis-cluster 172.18.0.2 6379
1:X 18 Jun 2020 21:25:08.580 # +switch-master redis-cluster 172.18.0.2 6379 172.18.0.4 6379
1:X 18 Jun 2020 21:25:08.581 * +slave slave 172.18.0.3:6379 172.18.0.3 6379 @ redis-cluster 172.18.0.4 6379
1:X 18 Jun 2020 21:25:08.581 * +slave slave 172.18.0.2:6379 172.18.0.2 6379 @ redis-cluster 172.18.0.4 6379
That’s all folks, as you can see we have implemented a spring boot redis app and learned how to create different redis networks.
If you have any questions or feedback don’t hesitate to write your thoughts in the comments/responses section.
For issues related to code, please feel free to create an issue directly in GitHub repository.
https://github.com/avinash10584/spring-boot-redis-cluster
Thank you for reading! If you enjoyed it, please clap 👏 for it.