当前位置：网站首页>Docker implements redis cluster mode hash slot partition for 100 million level data storage

A single machine can never complete , Distributed storage must be used . The problem lies in redis How to land , Let's assume that there is 5 Taiwan machine , Which one should exist when the data comes ？ Can you ensure that the machine reading data is correct ？

Solution

One 、 Hash remainder partition

2 One hundred million records are 2 One hundred million k,v, We can't do it on a single machine. We have to distribute multiple machines , Suppose there is 3 Machines form a cluster , Every time a user reads or writes, he or she uses a formula ：hash(key) % N Number of machines , Calculate the hash value , It is used to determine which node the data is mapped to . give an example ： Take remainder as 0, Here it is stored in the first , by 1 Store in the second , by 2 Store in the third .Redis Medium key Generally, it is not repeated , Using the same set of hash algorithm can ensure that you store data and read data on the same machine .

advantage ：

Simple and crude , Just estimate the data and plan the nodes , for example 3 platform 、8 platform 、10 platform , Can guarantee a period of data support . Use Hash The algorithm makes a fixed part of the requests fall on the same server , In this way, each server will process a part of the requests （ And maintain the requested information ）, Load balancing + The role of divide and rule .

shortcoming ：

The originally planned node , It is troublesome to expand or shrink the capacity , Regardless of expansion or contraction , Every time the data changes, the node changes , The mapping relationship needs to be recalculated , There is no problem when the number of servers is fixed , If elastic capacity expansion or fault shutdown is required , The original mold formula will change ：Hash(key)/3 Will become Hash(key) /?. At this time, the result of the remainder operation of the address will change greatly , The server obtained according to the formula will also become uncontrollable .
Some redis The machine is down , Due to the change in the number of sets , It can lead to hash Take all the remaining data and shuffle again .

In reality, it is inevitable to expand and shrink the capacity of machinery and equipment , Therefore, this scheme is not recommended .（ Small factories can use ）

Two 、 Consistent hash algorithm partition

Distributed cache data change and mapping problem . If a machine goes down , The number of denominators changes , There will be problems with natural remainder . Propose consistency Hash Solution , The purpose is when the number of servers changes , Minimize the mapping relationship between client and server . There are three steps , as follows ：

1. The algorithm constructs a consistent hash ring

A consistent hash algorithm must have hash Function and generate... According to the algorithm hash value , All possible hash values of this algorithm will form a full set , This set can become a hash Space [0,2^32-1], This is a linear space , But in the algorithm , We connect it end to end through appropriate logical control (0 = 2^32), This makes it logically form an annular space .
It is also based on the method of using remainder , The node remainder method introduced in the previous notes is for nodes （ The server ） Take the remaining quantity . And consistency Hash The algorithm is right 2^32 Remainder , Simply speaking , Uniformity Hash The algorithm organizes the whole hash space into a virtual circle , Suppose a hash function H The value space of is 0-2^32-1（ That is, the hash value is a 32 Bit unsigned shaping ）, The whole hash ring is shown in the figure below ： The whole space is organized in a clockwise direction , The point directly above the circle represents 0,0 The first point on the right represents 1, And so on ,2、3、4、…… until 2^32-1, in other words 0 The first point on the left represents 2^32-1, 0 and 2^32-1 The direction coincides in zero , We put this by 2^32 The circle of dots is called Hash Ring .

In a nutshell , What we usually call collection is like a line , It's straight . But here , We broke him into a circle , Let it close end to end . This ring is called hash ring . Then everything will be stored in this ring .

2. The server IP Node mapping

Connect all nodes in the cluster IP The node is mapped to a position on the ring .
Hash each server , You can choose the server IP Or the host name as a key to hash , In this way, each machine can determine its position on the hash ring . If 4 Nodes NodeA、B、C、D, after IP Hash function calculation of address (hash(ip)), Use IP The location of the address hash in the ring space is as follows ：

3.key Drop key rules for server

When we need to store a kv Key to time , First calculate key Of hash value ,hash(key), Put this key Use the same function Hash Calculate the hash value and determine the position of this data on the ring , Clockwise along the ring from this position “ walk ”, The first server encountered is the server it should be located to , And store the key value pair on the node .
If we have Object A、Object B、Object C、Object D Four data objects , After hashing , The position in the ring space is as follows ： According to consistency Hash Algorithm , data A Will be set to Node A On ,B Set to Node B On ,C Set to Node C On ,D Set to Node D On .

advantage ：

Fault tolerance of consistent hash algorithm

hypothesis Node C Downtime , You can see the object at this time A、B、D Not affected , Only C The object is repositioned to Node D. General , In consistency Hash In the algorithm, , If a server is not available , Then the affected data is only from this server to the previous server in its ring space （ I.e. the first server I met when I walked in a counterclockwise direction ） Data between , Others will not be affected . In short , Namely C Hang up , What is affected is B、C Data between , And the data will be transferred to D For storage .

Scalability of consistency FACHE Algorithm

The amount of data has increased , Need to add a node NodeX,X The position of is A and B Between , What is affected is A To X Data between , Put the A To X Enter your data into X You can go up. , Will not lead to hash Take all the remaining data and shuffle again .

shortcoming ：

Data skew problem of consistent hash algorithm

Uniformity Hash When the number of service nodes is too small , It is easy to cause data skew due to uneven node distribution （ Most of the cached objects are cached on a certain server ） problem , For example, there are only two servers in the system ：

To put it bluntly, if there are too few equipment , It will lead to uneven storage .（ Small factories cannot be used ）

3、 ... and 、 Hash slot partition

The hash slot is essentially an array , Array [0,2^14 -1] formation hash slot Space . It can solve the problem of uniform distribution , Another layer is added between data and nodes , Call this floor Hashi trough （slot）, Used to manage the relationship between data and nodes , Now it's equivalent to a slot on the node , There's data in the slot .

This is actually a little similar to Java in ,Service The layer calls a DAO Interface , To complete the addition, deletion, modification and query of data .

The groove solves the problem of granularity , It's equivalent to increasing the granularity , This makes it easy to move data .
Hash solves the mapping problem , Use key The hash value to calculate the slot , Easy data distribution .

A cluster can only have 16384 Slot , Number 0-16383（0-2^14-1）. These slots are allocated to all primary nodes in the cluster , The allocation strategy does not require . You can specify which numbered slots are assigned to which master node . The cluster will record the corresponding relationship between nodes and slots . After solving the relationship between node and slot , Next we need to be right key Find the hash value , Then on 16384 Remainder , What is the remainder key Fall into the corresponding slot .slot = CRC16(key) % 16384. Move data in slots , Because the number of slots is fixed , It's easier to deal with , So the problem of data mobility is solved .

Hash slot calculation

Redis There's a built-in... In the cluster 16384 Hash slot ,redis The hash slot will be mapped to different nodes approximately equally according to the number of nodes . When need is in Redis Place one in the cluster key-value when ,redis First pair key Use crc16 The algorithm works out a result , Then get the result right 16384 Mod , So each of them key They all have a number in 0-16383 The Hashi trough between , That is, mapping to a node . The following code ,key And A 、B stay Node2, key And C Fall in the Node3 On .

there [0-5460] The slot range will be tested later ！

    @Test
    public void test1(){
        System.out.println(SlotHash.getSlot("A"));//6376
        System.out.println(SlotHash.getSlot("B"));//10374
        System.out.println(SlotHash.getSlot("C"));//14503
        System.out.println(SlotHash.getSlot("van"));//866
    }

3 Lord 3 from Redis Cluster expansion and contraction configuration

One 、 newly build 6 individual docker Container instance

Start the container , Six in all , Three masters and three followers ：

docker run -d --name redis-node-1 --net host --privileged=true -v /data/redis/share/redis-node-1:/data redis:6.0.8 --cluster-enabled yes --appendonly yes --port 6381

docker run -d --name redis-node-2 --net host --privileged=true -v /data/redis/share/redis-node-2:/data redis:6.0.8 --cluster-enabled yes --appendonly yes --port 6382

docker run -d --name redis-node-3 --net host --privileged=true -v /data/redis/share/redis-node-3:/data redis:6.0.8 --cluster-enabled yes --appendonly yes --port 6383

docker run -d --name redis-node-4 --net host --privileged=true -v /data/redis/share/redis-node-4:/data redis:6.0.8 --cluster-enabled yes --appendonly yes --port 6384

docker run -d --name redis-node-5 --net host --privileged=true -v /data/redis/share/redis-node-5:/data redis:6.0.8 --cluster-enabled yes --appendonly yes --port 6385

docker run -d --name redis-node-6 --net host --privileged=true -v /data/redis/share/redis-node-6:/data redis:6.0.8 --cluster-enabled yes --appendonly yes --port 6386

The effect of pulling the image for the first time is as follows ：

Do it all ：

Remember after running ps Check it out. , To make sure there are no problems ：

docker ps

The step-by-step explanation of the above code is as follows ：

docker run： Create and run docker Container instance
--name redis-node-6： Container name
--net host： Using the host IP And port , Default
--privileged=true： Get host root User permissions
-v /data/redis/share/redis-node-6:/data： Container data volume
redis:6.0.8：redis Image and version number
--cluster-enabled yes： Turn on redis colony
--appendonly yes： Turn on persistence
--port 6385：redis Port number

Two 、 Into the container redis-node-1 And for 6 Build cluster relationship for each device

1. Into the container

docker exec -it redis-node-1 /bin/bash

2. Build a master-slave relationship

Here you need to pay attention to your truth IP Address ：

redis-cli --cluster create 192.168.150.30:6381 192.168.150.30:6382 192.168.150.30:6383 192.168.150.30:6384 192.168.150.30:6385 192.168.150.30:6386 --cluster-replicas 1

--cluster-replicas 1 For each master Create a slave node , Execution effect ：

At the same time, there is a slot range display , For example, for unit 1 [0-5460], Focus on , I have to take a test later ！

Input yes And return ：

The master-slave allocation is completed .

3、 ... and 、 With 6381 As an entry point , View the cluster status

View node status ：

redis-cli -p 6381

CLUSTER info

CLUSTER nodes You can clearly see who is the master and who is the slave

CLUSTER nodes

Master slave failover migration

One 、 Data read / write storage

At this point we go into redis Write some data

1. adopt exec Get into

redis-cli -p 6381

2. Yes 6381 Add a few more key

casual set Order something in

set k1 v1
set k2 v2
set k3 v3
set k4 v4

Storage error . The reason is because he It is beyond the range of single machine slot （ The test sites mentioned above ）. Let's look back at the slot range we saw when allocating master-slave , Machine one is [0-5460], And here is beyond , Even reached 6000+. So it can't be saved . It's not even a cluster if you can't save it ？

The reason is that there is a problem with our startup method , It can no longer be used here redis-cli -p 6381 This way to connect （ This is the connection mode of a single machine ）, This method is not suitable for cluster environment . The correct method is as follows

3. Prevent route failure and add parameters -c

exit sign out

exit

Reconnect the

redis-cli -p 6381 -c

FLUSHALL Clear it out

FLUSHALL

Then we go back to set data

set k1 v1

Deposit successful , It can be seen here that it has been saved 12706 Slot No , We mentioned above , The slot range of unit 1 is only [0-5460], After passing the algorithm here , Successfully saved the data to machine 3 .

Redirected to slot[12706]： Has been redirected to 12706 Slot No

After this order, we found ourselves jumping to 6383 node , Let's try saving some more data

set k2 v2

Successfully jump back 6381

set k3 v3

If the storage slot is within the slot range of the machine , Will not jump , Direct storage

set k4 v4

Here we jump to 6382, The slot here [8455] You can compare with the previous Slot range Take a look at the screenshot , It's stored on machine two . How to view the slot range is shown in the following section

4. View cluster information

Pay attention to change your reality here IP

redis-cli --cluster check 192.168.150.30:6381

Two 、 Failover migration

1. Downtime test

Here's a demonstration , When master After downtime ,slave How to switch upper

Let's first look at the master-slave relationship of our equipment , Check after connecting

redis-cli -p 6381
CLUSTER nodes

Here you can know my 6381 The slave is 6385

then 6381 Break down , On the host docker stop once

docker stop redis-node-1

We'll... After we stop ps Check it out.

docker ps

Can be determined , Machine one has been stopped （ It's down. ）, At this time, we will pass through the second machine , That is to say 6382 Go up and check the status information

docker exec -it redis-node-2 /bin/bash
redis-cli -p 6382 -c

CLUSTER nodes

You can see 6381 Hang up ,6385 Succeed in getting on top , from slave Turned into master

get Here's the data , See if you can find

get k1
get k2
get k3
get k4

Data search succeeded , It's obvious that when you hang up, you hang up , It doesn't affect my reading data

2. To restore

Let's see , If 6381 raised , So it's with 6385 Whether the master-slave relationship will change

Start the container

docker start redis-node-1 
docker ps

Then check the status information

CLUSTER nodes

It can be seen that 6381 Keep it after resurrection slave,6385 Is still master, If you want to 6381 When back again master, Then you just need to 6385 Stop , Then restart it （ It is not recommended here docker restart, The suggestion is first stop Again start）

docker stop redis-node-5

CLUSTER nodes

Start again 6385

docker start redis-node-5

CLUSTER nodes

You can check it again

redis-cli --cluster check 192.168.150.30:6381

You can see that the master-slave restore is successful

Master slave capacity expansion

If our 3 Lord 3 I can never bear the current demand , Add two more devices into the cluster （6387 and 6388）, And you need to 6388 Set to 6387 Of slave How to operate ？ And after joining , How to allocate the hash slots that have been divided before ？

One 、 newly build 6387 and 6388 Two nodes

docker run -d --name redis-node-7 --net host --privileged=true -v /data/redis/share/redis-node-7:/data redis:6.0.8 --cluster-enabled yes --appendonly yes --port 6387

docker run -d --name redis-node-8 --net host --privileged=true -v /data/redis/share/redis-node-8:/data redis:6.0.8 --cluster-enabled yes --appendonly yes --port 6388

Develop good habits , Remember ps Check whether it is normal

docker ps

8 All nodes are normal

Two 、 Get into 6387 Inside the container instance

docker exec -it redis-node-7 /bin/bash

3、 ... and 、 Will add 6387 As master Nodes join the cluster

Pay attention to fill in your authenticity here IP, To put it bluntly, it means 6387 Look for 6381, recognize 6381 Be the boss , Then join the cluster by yourself .

redis-cli --cluster add-node 192.168.150.30:6387 192.168.150.30:6381

Node added successfully , But what can be seen here is still 3 Lord 3 from , Let's move on

Four 、 View cluster information -- for the first time

redis-cli --cluster check 192.168.150.30:6381

You can see... Here 6387 Have done as master Joined the cluster , however 6387 The slot has not been allocated yet

5、 ... and 、 Reallocate slots

reshard It means to reassign the slot number

redis-cli --cluster reshard 192.168.150.30:6381

Here we will ask how much to allocate , Input 4096 that will do , Why is 4096 Well ？ because ：

16384/4（master Number ） Distribute evenly .. So it is 4096, Then I'll ask where you put it ID Inside , Copy 6387 Of ID that will do

Then the input all And then go back , Reshuffle , Assign slot number

You need to yes once , You can enter.

The slot number is reassigned

6、 ... and 、 View cluster information -- The second time

Pay attention to reality IP

redis-cli --cluster check 192.168.150.30:6381

Here you can see

6381 The range is [0-5460], Now it's [1365-5460]
6382 The range is from [5461-10922] Turned into [6827-10922]
6383 The range is from [10923-16383] Turned into [12200-16383]

You can see all master The range of the right section of the equipment has not changed . To put it bluntly, the cost of redistribution is too high , So the front three master Are divided into a part （1364 individual ） Come out and give it to 6387. Not all are redistributed evenly .

7、 ... and 、 Master node 6387 Assign slave nodes 6388

take 6388 As slave Hang up 6387 above . Pay attention to reality IP, hinder ID by 6387 Of ID, Copy and paste from above

redis-cli --cluster add-node 192.168.150.30:6388 192.168.150.30:6387 --cluster-slave --cluster-master-id 08c0499f4e03063cc3ae903f74cdf0c2de089cf3

Distribution succeeded

8、 ... and 、 View cluster information -- third time

redis-cli --cluster check 192.168.150.30:6381

4 Lord 4 Configuration succeeded

Master slave volume reduction

4 Lord 4 from ---->3 Lord 3 from , take 6387 and 6388 Delete . The deleted slot conforms to the allocation ？ Delete first master6387 Or delete it first slave6388, What is the order of the whole ？

The answer to the next question is to delete the slave first slave6388, Reassign the cleared slots to the first three machines , Finally, delete the host master6387.

One 、 Check the cluster and get 6388 The node of ID

redis-cli --cluster check 192.168.150.30:6381

Inside the box in the figure is 6388 The node of ID

Two 、 take 6388 Delete

Pay attention to reality IP And nodes ID

redis-cli --cluster del-node 192.168.150.30:6388 e8a034707acff6572fa0646514b6d9f773d62efe

Here you can check again ,6388 It's gone , And then there were 4 Lord 3 from

redis-cli --cluster check 192.168.150.30:6381

3、 ... and 、 take 6387 The slot of is empty Redistribution

Slots are allocated here to 6381 Redistribute as a foothold

redis-cli --cluster reshard 192.168.150.30:6381

The specific operation is shown in the following figure

middle yes And then go back

Four 、 Check the cluster status information

redis-cli --cluster check 192.168.150.30:6381

Slot allocation succeeded

5、 ... and 、 Delete 6387