Codis 3. X expansion and contraction

Catalog

Codis 3.x edition

Codis Capacity expansion

Codis Shrinkage capacity

codis bgsave working principle

codis-admin Common commands

---

Codis 3.x edition

• newest release Version is codis-3.2,codis-server be based on redis-3.2.8
• Support slot Synchronous migration 、 Asynchronous migration and concurrent migration , Yes key There is no limit to size , The migration performance is greatly improved
• comparison 2.0： The whole cluster component communication mode is reconstructed ,codis-proxy And zookeeper Achieve decoupling , Abandoned codis-config etc.
• Metadata storage support etcd/zookeeper/filesystem etc. , Self expandable support for new storage , During normal cluster operation , Even if the meta storage fails, it will no longer affect codis colony , Greatly enhance codis-proxy stability
• Yes codis-proxy A lot of performance optimization has been done , By controlling GC frequency 、 Reduce object creation 、 Memory pre allocation 、 introduce  cgo、jemalloc  etc. , Make it throughput or delay , Have reached codis The best in the project
• proxy Realization select command , Support more DB
• proxy Support for read/write separation 、 Priority reading is the same as IP/ Same as DC Next copy function
• be based on redis-sentinel Realize automatic switchover between active and standby
• Implementation dynamics pipeline Buffer zone （ Reduce memory allocation and the resulting GC problem ）
• proxy Supported by HTTP Request real-time access runtime metrics, Easy to monitor 、 Operation and maintenance
• Supported by influxdb and statsd collection proxy metrics
• slot auto rebalance Algorithm from 2.0 Based on the max memory policy The change is based on group Next slot Number
• Provides a more friendly dashboard and fe Interface , Added a lot of buttons 、 Jump Links 、 Error status, etc , It is conducive to rapid discovery 、 Handle cluster failures
• newly added  SLOTSSCAN  Instructions , It is easy to obtain the data of each cluster slot All under key
• codis-proxy And codis-dashbaord Support docker Deploy

Codis Capacity expansion

Theory and deployment codis It's no big difference , There are some small things to be aware of

codis Capacity can be used to calculate ： If the single instance is 8G Suppose we have 3 individual group

be 3 * 8G = 24GB

group Number    *  Single instance allocation memory  =  Available capacity

There can be multiple instances in a group , The relationship is a master many slave Pattern , The default read / write is from master ,slave Only responsible for synchronization master The data of , When master After hanging up, re-election through the sentry master Switch .

Expansion machine

The server	System	edition
10.136.47.3	Centos7	3.2.11
10.103.17.50	Centos7	3.2.11

1.  land 10.126.174.24(ansiable) The server , Move to the directory

cd /home/sunwenbo/codis3-capacity/ansible-codis/roles

2. modify ip.txt, Servers to be expanded ip Write to this file . Keep it here server 、proxy as follows

[dashboard] [proxy] 10.136.47.3 10.103.17.50 [server] 10.136.47.3 10.103.17.50 [sentinel]

3. modify /home/sunwenbo/codis-capacity/ansible-codis/roles/vars/vars.yml file

  Expand the source code

client-output-buffer-limit normal	client-output-buffer-limit slave	client-output-buffer-limit pubsub
For normal clients , Limit to 0, There is no limit . Because the normal client usually adopts the blocking message reply mode , What is blocking ？ Such as ： Send a request , Waiting to return , Resend request , Wait for the return . In this mode , Usually it doesn't lead to Redis Server output buffer heap expansion ;	about slave For the client , The size limit is 256M, The persistence limit is when the client buffer size persists 60 Seconds more than 64M, Then close the client connection .	about Pub/Sub client （ That is to say, release / A subscription model ）, The size limit is 8M, When the output buffer exceeds 8M when , Will close the connection . The persistence limit is , When the client buffer size continues 60 Seconds more than 2M, Then close the client connection ;

redis Of client output buffer limit Can be used to forcibly disconnect those that cannot be accessed from redis The server reads data Fast enough client . A common case is Pub/Sub in , The speed of consumers cannot match that of message producers Match the speed .client output buffer limit It can be applied to three types of clients ：

• Ordinary client, Include monitor
• slave Used to synchronize master Data client
• Pub/Sub At least one... Is subscribed to in the schema channel Or pattern client

The syntax for this configuration is :

client-output-buffer-limit <class> <hard limit> <soft limit> <soft seconds>

 Connect to redis server The client of , If it cannot consume the data returned to it by the fast consumption server , Its presence server  The memory occupied by the client (output buffer size) More and more , Once the size exceeds hard limit,  Or more than  soft limit A span ,server The client will immediately disconnect the client , And release the resources occupied by the client .  for example , if hard limit Configure to 32M, soft limit Configure to 10M/10s, once output buffer Use   Of memory has reached 30M, Or more than in a row 10M Time maintenance of 10s, Then the client connection will be immediately disconnected .

By default , For ordinary clients output buffer size No restrictions , because server Only in its request Data will be sent when ; Only when the asynchronous client requests data faster than it receives , Will produce out buffer size The problem of inflation . For use as pubsub and slave The client of , because server Will actively push data to they , Need to set up output buffer size The limitation of .

It can be done by putting hard limit/soft limit Set to 0 To disable the configuration item , Such as ：

client-output-buffer-limit normal 0 0 0 client-output-buffer-limit slave 0 0 0 client-output-buffer-limit pubsub 32mb 8mb 60

4. Execute capacity expansion command ,ansiable No return error , The deployment is considered successful

ansible-playbook -i ip.txt  main.yml -b

5. land codis-fe The management page is used for capacity expansion configuration

•      5.1 Capacity expansion proxy node , Click on New Proxy Add newly deployed instances to the cluster
   

•      5.2 Capacity expansion server node , newly build group, After that, add the expanded nodes group in .  Be careful ： Instance join group The rules , Do not add instances on the same machine to one gourp in , Prevent server downtime , Instance cannot switch roles                    

 

          

6.  Manually adjust the instance role master、slave

    Click on Master The bottom color of the column is the small wrench behind the red instance

   

    At this time codis Will be carried out in master、slave The role assignment of , You can see examples 10.136.47.3:6381 Considered an example 10.103.17.50:6380 Of slave, in other words 10.103.17.50:6380 by master

    By default, the instance of the first row in the group will be selected as master, In order to provide high availability and high performance services , We should try to avoid master On the same machine . You can set the role of the instance through the page, as shown in the following figure

    Click on PROMOTE, You can make this instance master, You can see 10.136.47.3:6381 It has arrived group 5 The first line of .

7.  take group Servers in join codis colony , Distribute Slots

      By this time we have codis-server Deploy 、 establish group、 And set the server instance master Role assignment , The following needs to be expanded codis Instance join to codis colony , Realize capacity expansion

     Slots Distribute equally to each group, Capacity expansion is very simple. Just click on the page Rebalance All Slots Button codis Will be automatically allocated Slots 了 , You can see when it is not allocated , What we see is that codis take 1024 Slots are allocated to existing slots 3 individual group

     

    

    Click on Rebalance All Slots after codis You will find two new groups , And divide the de grooving position and assign it to group4、group5  

   

    Click on ok ,codis Specific slots will be allocated group, It will be distributed in a moment , The following figure is a screenshot after slot allocation , You can see that there is already 5 A group

   

  Then let's look at our new group, Data synchronization has started

  

  Perform the two steps shown in the figure below , take codis server Set to active , And click the SYNC To synchronize

  

  If you return to the following page, the expansion succeeds

   

Codis Shrinkage capacity

according to codis github describe codis 3.x The version has been abolished codis-config Management tools , Use Codis Built in commands can be used to slot Migration and Codis Dashboard Conduct management .

Related information  

https://github.com/CodisLabs/codis/blob/release3.2/doc/redis_change_zh.md

https://github.com/CodisLabs/codis/blob/release3.2/doc/tutorial_zh.md

Now we are expanding our capacity group 5 The corresponding two instances are deleted from the cluster

1.   codis-fe Manage pages to shrink

      Suppose we want to delete group 5, You can see the current group 5 The allocated slots are the two purple parts in the figure （codis Altogether 1024 Slots . 0  -  1023）

        

        need group 5 slots Data migration to other group , Appoint slot big and slot end to gourp After clicking on Migrate Range, Start the migration

  Now I see group The data of has been reduced by more than half . Next, there are purple ones left slot Slot migration to other gropu

  Because just now we will 888 - 1023 Of slot All moved to gourp1 Lead to group1 Memory usage is nearly full . Be careful ： It should be noted here that if we want to migrate slot Big words , Find a group with enough resources to store , In case of data loss .

  So put the non migrated slot Migrate to group2 

  Look at it again. gourp 5, There's no more data

2. Delete codis Examples and group 

  Delete the instance

  Delete the group   It should be noted that if volume reduction is required , Make sure that the corresponding group Of slot Migration , Otherwise delete group in master An error will be reported in the instance .

Here, the volume reduction is completed

codis bgsave working principle

Related references https://www.cnblogs.com/evakang/p/12078919.html

We put Redis It's divided into three parts , They are clients 、 Master and slave nodes , If the slave node wants to synchronize the data of the master node , It starts with Sync Instructions to the master node , The master node will execute after receiving the instruction BGSAVE Command to generate RDB file , This RDB A file is a snapshot file , It is Redis One of the two backup methods , The other is called AOF, Its principle is to store all the write instructions in the file ,mysql Of binlog The principle is the same .

If the master node is generating RDB In the process , The client sent a write instruction , At this time, the master node will write all instructions to the buffer , etc. RDB Finished generating , Will be able to RDB The file is sent to the slave node , Finally, the instruction of buffer is sent to the slave node . This completes the whole copy .

We just said that it is flawed to be a mere master , The flaw is that if we want to store massive amounts of data , that BGSAVE Instruction generated RDB The file will be huge , The transfer of this file to the slave node is also very slow , If there are a lot of buffer commands , Synchronizing data from a node can also take a long time , therefore , To solve the problem of single point and mass storage , You still need to consider clustering .

codis-admin Common commands

View groups codis-admin -v --dashboard=bj2-e05-126-165-6-lg.yidian.com:18130 --list-group see codis proxy codis-admin -v --dashboard=bj2-e05-126-165-6-lg.yidian.com:18130 --list-proxy see codis Slot status codis-admin -v --dashboard=bj2-e05-126-165-6-lg.yidian.com:18130 --slots-status stop it dashboard service codis-admin -v --dashboard=bj2-e05-126-165-6-lg.yidian.com:18130 --shutdown Delete group The slot corresponding to the group codis-admin -v --dashboard=10.126.165.6:18130 --slots-assign --beg=512 --end=1023 --offline --confirm Delete the group codis-admin -v --dashboard=bj2-e05-126-165-6-lg.yidian.com:18130 --remove-group --gid=2