Redis pipeline technology / partition

Redis Pipeline technology

Redis Is a client based - Server models and requests / In response to the protocol TCP service . This means that the request will usually follow the following steps ：
The client sends a query request to the server , And listen for socket returns , Usually in blocking mode , Wait for the server to respond .
The server processes the command and returns the result to the client .
Redis Pipeline technology allows the client to continue sending requests to the server when the server does not respond , Read the responses of all servers for the last time .

example

To see redis The Conduit , Just start redis Instance and enter the following command ：

$(echo -en "PING\r\n SET runoobkey redis\r\nGET runoobkey\r\nINCR visitor\r\nINCR visitor\r\nINCR visitor\r\n"; sleep 10) | nc localhost 6379+PONG+OKredis:1:2:3

In the example above , We use ping Command check redis Is the service available . Then we will runoobkey Is set to redis, And then we get runoobkey Value , And automatically triple the number of visitors .
In the returned results , We can see that these commands are submitted to redis service , Read the responses of all servers for the last time
The most significant advantage of pipeline technology is to improve redis Service performance .
Some test data

require 'rubygems' require 'redis'def bench(descr) start = Time.now yield puts "#{descr} #{Time.now-start} seconds" enddef without_pipelining r = Redis.new 10000.times {     r.ping } enddef with_pipelining r = Redis.new r.pipelined {     10000.times {         r.ping     } } endbench("without pipelining") {     without_pipelining } bench("with pipelining") {     with_pipelining }

In the next test , We will use redis Of Ruby Client to support the characteristics of Pipeline Technology , And test the speed-up effect of assembly line technology .
In LAN Mac OS X The data of the above simple script executed on the system shows , After opening the pipeline , Round trip latency has improved significantly .

Redis

Partitioning is to divide data into multiple redis The process of example , Therefore, each instance only saves a subset of the key .
The advantages of zoning
By using the total memory of multiple computers , We can build a larger database .
Through multiple cores and computers , We can expand our computing power ; Through multiple computers and network adapters , We can expand the network bandwidth .
Insufficient partitions
redis Some functions of the do not perform well in partitioning ：
Typically, multiple key operations are not supported . for example , When two sets map to different redis When an instance , You cannot perform an intersection operation on these two sets .
Cannot use multiple keys Redis Business .
When using partitions , Data processing is more complex . for example , You need to deal with multiple rdb/aof file , And back up persistent files from multiple instances and hosts .
Adding or removing capacity is also complex . majority redis The cluster supports transparent data balancing for adding and deleting nodes at run time , But other systems （ Such as client partition and proxy ） This feature is not supported . However , A technique called pre hardening is helpful .

Partition type

Redis There are two types of divisions . Suppose there are four redis example R0、R1、R2、R3 And multiple keys represent the user , for example user:1 and user:2. For a given key , There are many different ways to choose which instance the key is stored in . let me put it another way , There are different systems that map keys to redis service .

Range partitioning

The simplest way to partition is by range , Mapping objects in a specific range to specific redis example .
for example ,ID by 0 To 10000 The user of will be saved to the instance R0,ID by 10001 To 20000 Users of will be saved to R1, And so on .
This method is feasible , It can be used in practice . The disadvantage is that there is a mapping table from interval range to instance . This table needs to be managed . meanwhile , It also requires mapping tables for various objects , It's not usually redis Good way .

Hash partition

Another partitioning method is hash partitioning . This applies to any key , And it doesn't need to be an object name ： This form is as simple as the following description ：
Use the hash function to convert the key to a number , for example , Use CRC32 hash function . Key in foobar On the implementation CRC32（foobar） Will output a similar 93024922 The integer of .
Take the modulus of this integer and convert it to 0 To 3 Number between , Then this integer can be mapped to four redis One example .93024922%4=2, This means that the key foobar It should be kept in R2 In the example . notes ： Modular operation is the remainder of division , In many programming languages % Operator implementation .