Sersync/lsync real-time synchronization

Chapter one Why need real-time synchronization

1. What problems are solved by real-time synchronization in the second stage :

solve NFS Single point problem
Backup NFS Data and provide redundant service functions

2. Real time synchronization difficulties

 What conditions are synchronized 
 How often do I sync 
 What backup method is used to synchronize 
 What tools are used to synchronize 

Chapter two inotify brief introduction

Inotify It's a powerful , Fine grained , Asynchronous file system event monitoring mechanism 
Linux2.6.13 Joined since inotify Support , adopt inotify Can monitor the file system to add , Delete , modify , Mobile and other events 
 Using this kernel interface , Third party software can monitor the changes of files in the file system 
 and inotify-tools It's the software that implements this kind of monitoring 

The diagram

The third chapter Inotify Real time synchronization

1. Check whether the current system supports inotify

[[email protected] ~]# uname -r
3.10.0-862.el7.x86_64
[[email protected] ~]# ls -l /proc/sys/fs/inotify/
total 0
-rw-r--r-- 1 root root 0 Oct 21 10:47 max_queued_events
-rw-r--r-- 1 root root 0 Oct 21 10:47 max_user_instances
-rw-r--r-- 1 root root 0 Oct 21 10:47 max_user_watches
===== ===== ===== ===== ===== ===== ===== ===== ===== ===== ===== ===== ===== =====
max_queued_events   =====> Set up inotify Case Events (event) The number of events that the queue can hold 
max_user_instances  =====> Set what each user can run inotify perhaps inotifywatch Number of command processes 
max_user_watches    =====> Set up inotifywait perhaps inotifywatch The number of files that the command can monitor ( Single process )

2. install inotify-tools

[[email protected] ~]# yum install inotify-tools -y

3.inotifywait Detailed parameter explanation

inotifywait Parameter description 
 Parameter name              Parameter description 
-m ,-monitor        Always keep the event listening state 
-r,-recursive       Recursive query Directory 
-q,-quiet           Print only information about monitoring events 
-exclude            When excluding files or directories , Case insensitive 
-t,-timeout         Timeout time 
--timefmt           Specify the time output format 
--format            Specify the output format 
-e,event            Specify add later , Delete , Change and other events 
inotifywait events  Event description 
access              Read the contents of a file or directory 
modify              Modify the contents of files or directories 
attrib              The properties of a file or directory change 
close_write         Modify the content of the real file 
close_nowrite   
close   
open                File or directory opened 
moved_to            Move files or directories to 
moved_from          Files or directories from ... Move to 
move                Move files or directories to the monitoring directory 
create              Create a file or directory under the monitoring directory 
delete              Delete the file or directory under the monitoring directory 
delete_self 
umount              Uninstall file system 

4. see inotify Default parameters

[[email protected] ~]# sysctl -a |egrep max_queued_events
fs.inotify.max_queued_events = 16384
[[email protected] ~]#  sysctl -a |egrep max_user_watches
fs.epoll.max_user_watches = 201256
fs.inotify.max_user_watches = 8192
[[email protected] ~]# sysctl -a |egrep max_user_instances
fs.inotify.max_user_instances = 128

5. Test the above command

Parameter interpretation :

%T   # Call and display the defined time format 
%w   # Show the absolute path of the changed file 
%f   # Display the monitored file name , After removal, only the directory path is displayed 
-e delete,create   # Specify the type of monitoring file changes  

1. Open two windows : test create and delete

inotifywait -mrq --timefmt '%d/%m/%y %H:%M' --format '%T %w%f' -e delete,create /backup

2. test close_write

inotifywait -mrq --timefmt '%d/%m/%y %H:%M' --format '%w%f' -e close_write /backup

3. Print out the event type

inotifywait -mrq --format '%w%f %e' -e delete,create /backup

6. Monitor the client to the situation rsync Push changed files

#!/bin/bash 

Path=/data
backup_Server=172.16.1.41
export RSYNC_PASSWORD=123456

/usr/bin/inotifywait -mrq --format '%w%f' -e create,close_write,delete /data | while read line
do
    echo ${line}
    if [ -f ${line} ]
    then
       rsync -az ${line} [email protected]${backup_Server}::data
    else
       rsync -az --delete /data/ [email protected]${backup_Server}::data
    fi
done

7.inotify Advantages and disadvantages

inotify advantage 
1) Monitoring changes in file system events 

inotify shortcoming 
1) If concurrency is greater than 200 Individual article (10-100k), Synchronization will be delayed 
2) Our previous script , It's all pushed once every time , But it is incremental , You can also synchronize only changing files 
3) After monitoring the incident , call rsync Synchronization is a single process ( Add & Concurrent )
4) The robustness of our own script is not high 

Chapter four sersync brief introduction

1.sersync Introduce

The following is an excerpt sersync Description of the author's official website

sersync Mainly used for server synchronization ,web Mirror image and other functions .
 be based on boost1.41.0,inotify api,rsync command Development .
 The most common synchronization solution currently used is inotify-tools+rsync

2.sersync advantage

1.sersync It's using c++ To write , And right linux The temporary files generated by the system file system and repeated file operations are filtered ( See Appendix for details , This filter script is not implemented ), So combine rsync When syncing , Save the running time and network resources , So faster 
2. Compare the above two items ,sersync It's easy to configure , among bin There are basically static compilers in the directory 2 Base file , coordination bin In the catalog xml The configuration file can be used directly .
3. In addition, this project is compared with other script open source projects , Use multithreading for synchronization , Especially when synchronizing large files , It can ensure that multiple servers remain synchronized 
4. This project has an error handling mechanism , Resynchronize the failed file through the failure queue , If it still fails , Then every 10 Two hours to resynchronize failed files 
5. The project comes with it crontab function , Just in xml Open... In the configuration file , According to your requirements , Every once in a while , No additional configuration crontab function 
6. This project socket And http Plug-in development , Meet your secondary development needs .

The fifth chapter sersync Installation and deployment

1. Hosting plan

nfs-server 172.16.1.31 rsync+inotify+sersync  nfs-client
backup     172.16.1.41 rsync-server           nfs-server

2. install sersync

cd /opt/
wget http://down.whsir.com/downloads/sersync2.5.4_64bit_binary_stable_final.tar.gz
tar zxf sersync2.5.4_64bit_binary_stable_final.tar.gz
mv GNU-Linux-x86 sersync
cd sersync
cp confxml.xml confxml.xml.bak

3. Configuration interpretation

# Directories to be monitored       <localpath watch="/data">   
# long-range rsync Server's IP Address and module name    <remote ip="172.16.1.41" name="data"/>  
#rsync Parameters of transmission     <commonParams params="-az"/> 
#rsync Password file path   <auth start="true" users="rsync_backup" passwordfile="/etc/rsync.passwd"/>    #

4. Configuration file modification

 23     <sersync>
 24         <localpath watch="/data">
 25             <remote ip="172.16.1.41" name="data"/>
 26             <!--<remote ip="192.168.8.39" name="tongbu"/>-->
 27             <!--<remote ip="192.168.8.40" name="tongbu"/>-->
 28         </localpath>
 29         <rsync>
 30             <commonParams params="-az"/>
 31             <auth start="true" users="rsync_backup" passwordfile="/etc/rsync.passwd"/>

5. See help notes

[[email protected] /opt/sersync]# ./sersync2 -h
set the system param
execute：echo 50000000 > /proc/sys/fs/inotify/max_user_watches
execute：echo 327679 > /proc/sys/fs/inotify/max_queued_events
parse the command param
_______________________________________________________
 Parameters -d: Enable daemon mode 
 Parameters -r: Before monitoring , Use monitoring directory with remote host rsync Order push once 
c Parameters -n:  Specify the number of daemons to turn on , The default is 10 individual 
 Parameters -o: Specify profile , By default confxml.xml file 
 Parameters -m: Enable other modules separately , Use  -m refreshCDN  Opening refresh CDN modular 
 Parameters -m: Enable other modules separately , Use  -m socket  Turn on socket modular 
 Parameters -m: Enable other modules separately , Use  -m http  Turn on http modular 
 No addition -m Parameters , The synchronization program is executed by default 
________________________________________________________________
[[email protected] /opt/sersync]# 

6. Start the service

./sersync2 -rdo confxml.xml

7. Test whether the data is synchronized

backup Server operations ：

1 Check it in seconds /data Number of files in the directory

cd /data
while true ;do ls |wc -l;sleep 0.1;done

nfs Server operation ：

cd /data
for i in {1..1000};do echo "${i}"; echo "${i}" > ${i}.txt;sleep 0.1;done

Chapter six lsyncd Sync

1. Official address

https://github.com/axkibe/lsyncd](https://github.com/axkibe/lsyncd

2.lsyncd Introduce

Lysncd  It's actually lua Language encapsulates  inotify  and  rsync  Tools , Adopted  Linux  kernel （2.6.13  And later ） Inside  inotify  Trigger mechanism , And then through rsync Desynchronization , Achieve real-time results .
 Its most commendable feature is , A perfect solution  `inotify + rsync` Massive file synchronization brings the problem of frequently sending file lists  ——  Through time delay or cumulative trigger event times .
 in addition , Its configuration is very simple ,lua Itself is a configuration language , Very readable .lsyncd There are also many working modes to choose , Local directory cp, Local directory rsync, Remote directory rsyncssh.
 Realize simple and efficient local directory synchronous backup （ Network storage is also mounted as a local directory ）, One command is done .

3.lsyncd install

yum install lsyncd -y

4. create profile

Monitor only 1 A catalog

[[email protected] ~]# cat /etc/lsyncd.conf
settings {
  logfile = "/var/log/lsyncd/lsyncd.log",
  statusFile = "/var/log/lsyncd/lsyncd.status",
  inotifyMode = "CloseWrite",
  maxProcesses = 8,
}
sync {
  default.rsync,
  source = "/data",
  target = "[email protected]::data",
  delete = true,
  exclude = { ".*" },
  delay = 1,
  rsync = {
    binary = "/usr/bin/rsync",
    archive = true,
    compress = true,
    verbose = true,
    password_file = "/etc/rsync.passwd",
    _extra = {"--bwlimit=200"}
  }
}

monitor 2 A catalog

[[email protected] ~]# cat /etc/lsyncd.conf
settings {
  logfile = "/var/log/lsyncd/lsyncd.log",
  statusFile = "/var/log/lsyncd/lsyncd.status",
  inotifyMode = "CloseWrite",
  maxProcesses = 8,
}
sync {
  default.rsync,
  source = "/data",
  target = "[email protected]::data",
  delete = true,
  exclude = { ".*" },
  delay = 1,
  rsync = {
    binary = "/usr/bin/rsync",
    archive = true,
    compress = true,
    verbose = true,
    password_file = "/etc/rsync.passwd",
    _extra = {"--bwlimit=200"}
  }
}

sync {
  default.rsync,
  source = "/backup",
  target = "[email protected]::backup",
  delete = true,
  exclude = { ".*" },
  delay = 1,
  rsync = {
    binary = "/usr/bin/rsync",
    archive = true,
    compress = true,
    verbose = true,
    password_file = "/etc/rsync.passwd",
    _extra = {"--bwlimit=200"}
  }
}

5. Profile explanation

Refer to netizens' blogs

https://www.cnblogs.com/zxci/p/6243574.html

6. Start command

systemctl start lsyncd 

7. Check and test

backup test

cd /data
while true ;do ls |wc -l;sleep 0.1;done

NFS test

cd /data
for i in {1..1000};do echo "${i}"; echo "${i}" > ${i}.txt;sleep 0.1;done