Rsync+inotify remote synchronization

rsync Introduce

1. rsync（Remote Sync, Remote synchronization ） Is an open source fast backup tool , It can mirror and synchronize the whole directory tree between different hosts , Support incremental backup , And keep links and permissions , And the optimized synchronization algorithm is used , Perform compression before transmission , So it is very suitable for remote backup 、 Image server and other applications
2. rsync The website of our official website is http://rsync.samba.org/, The latest version is
3. 1.3, from Wayne Davison For maintenance . As one of the most commonly used file backup tools ,rsync Tend to be Linux and UNIX One of the basic components installed by default

rsync The downside / Pull process

1. In the remote synchronization task , Responsible for initiating rsync The client of the synchronous operation is called the initiator , And is responsible for responding to requests from clients rsync The server that synchronizes is called the synchronization source .
2. During synchronization , The synchronization source is responsible for providing the original location of the file , The initiator should have read access to this location

To configure rsync The source server （192.168.80.11）

Turn off firewall and security features

systemctl stop firewalld.service 
setenforce 0

install rsync software package

rpm -q rsync
yum -y install rsync

establish /etc/rsyncd.conf The configuration file

vim /etc/rsyncd.conf      		# Add the following configuration 

uid = nobody					
gid = nobody					
use chroot = yes				# Locked in the source directory  
address = 192.168.80.11		    # Monitor address 
port 873                     	# Listening port  tcp/udp 873, It can be done by  cat /etc/services | grep rsync  see  
log file = /var/log/rsyncd.log	# Log file location  
pid file = /var/run/rsyncd.pid	# Store process ID File location of  
hosts allow = 192.168.80.0/24	# The client address allowed to access 
[wwwroot]                                                               
path = /var/www/html     		# The actual path to the source directory  
comment = Document Root of www.gxd.com	
read only = yes             	# Is it read-only  
dont comperss = *.gz *.bz2 *.tgz *.zip *.rar *.z  	# File types that are no longer compressed during synchronization  
auth users = backuper           # Authorized account , Multiple accounts are separated by spaces  
secrets file = /etc/rsyncd_users.db # Data file for storing account information 

# Such as anonymity , Just put one of them  “auth users”  and  “secrets file” The configuration item can be removed  

Create data file for backup account

vim /etc/rsyncd_users.db
backuper:264196			# There is no need to establish a system user with the same name 
chmod 600 /etc/rsyncd_users.db

Ensure that all users have access to the source directory /var/www/html Have read permission

mkdir -p /var/www/html
chmod +r /var/www/html/
ls -ld /var/www/html/

start-up rsync Service program

rsync --daemon
netstat -natp | grep rsync

close rsync service

kill $(cat /var/run/rsyncd.pid)
rm -rf /var/run/rsyncd.pid

Create files for testing

cd /var/www/html
echo "woshidashuaige" >> 1.txt

To configure rsync Initiator test backup （192.168.80.12）

Command format

rsync [ Options ]   Original location     Target location 

Common options	explain
-r	Recursive mode , Contains all the files in the directory and subdirectories
-l	For symbolic link files, still copy as symbolic link files
-v	Show details of the synchronization process
-z	Compress when transferring files
-a	Archiving mode , Recurse and keep object properties , Equate to -rlptgoD
-p	Keep the permission tag of the file
-t	Keep the time stamp of the file
-g	Keep the group tag of the file （ For super users only ）
-o	Keep the owner tag of the file （ For super users only ）
-H	Keep hard link files
-A	Retain ACL Attribute information
-D	Keep equipment files and other special files
–delete	Delete files that exist in the target location but not in the original location
–checksum	Decide whether to skip files based on the checksums of objects

Download the specified resource to the local /opt Backup under Directory

 Format 1 ：
rsync -avz [email protected]::wwwroot /opt/ # password 264196
 
 Format two ：
rsync -avz rsync://[email protected]/wwwroot /opt/

No interactive format configuration

cd /opt
rm -rf 1.txt

echo '264196' > /etc/1.pass  # Write a password file 
chmod 600 /etc/1.pass        # Give me read and write permission 

rsync -avz --delete --password-file=/etc/1.pass [email protected]::wwwroot /opt

Add to planned tasks

crontab -e
30 23 * * * /usr/bin/rsync -avz --delete --password-file=/etc/1.pass [email protected]192.168.80.11::wwwroot /opt/
 
systemctl restart crond
systemctl enable crond

rsync The upside / Push process

1. Linux Kernel from 2.6.13 The version begins to provide inotify Notification interface , Used to monitor changes in the file system , Such as file access 、 Delete 、 Move 、 Modify etc. . Using this mechanism , It is very convenient to realize file change alarm 、 Incremental backup , And respond to changes in directories or files in a timely manner
2. take inotify Mechanism and rsync Combination of tools , You can achieve triggered backup （ Real time synchronization ）—— As long as the original bit The set document changes , Start the incremental backup operation immediately ; Otherwise, it will be in a state of silent waiting , such , This avoids the latency when backing up according to a fixed cycle 、 The period is too dense and so on
3. because inotify The notification mechanism consists of linux Provided by kernel , Therefore, it is mainly used for local monitoring , It is more suitable for uplink synchronization when applied in triggered backup

To configure rsync The source server （192.168.80.11）

vim /etc/rsyncd.conf
...
read only = no       # Turn off read-only , Uplink synchronization requires writable permissions 
...

kill $(cat /var/run/rsyncd.pid)
netstat -natp | grep rsync
rsync --daemon
netstat -natp | grep rsync

chmod 777 /var/www/html/
ls -ld /var/www/html/

Configure initiator （192.168.80.12）

adjustment inotify Kernel parameters

cat /proc/sys/fs/inotify/max_queued_events
cat /proc/sys/fs/inotify/max_user_instances 
cat /proc/sys/fs/inotify/max_user_watches 

vim /etc/sysctl.conf 
fs.inotify.max_queued_events = 32768      # Monitor the time queue , The default is 16384
fs.inotify.max_user_instances = 1024	  # The maximum number of monitoring instances , The default is 128
fs.inotify.max_user_watches = 1048576	  # The maximum number of monitoring files per instance , The default is 8192
# When you want to monitor the directory 、 When the amount of file data is large or changes frequently , It is recommended to increase the parameter value 

sysctl -p

install inotify-tools

 use inotify The mechanism also needs to be installed  inotify-tools, To provide  inotifywait、inotifywatch  Auxiliary tool program .
notifywait： Can be monitored modify（ modify ）、create（ establish ）、move（ Move ）、delete（ Delete ）、attrib（ Property changes ） And so on , Output results as soon as there is a change .
inotifywatch： It can be used to collect file system changes , At the end of the run, output the change of the summary 

yum -y install gcc gcc-c++ 
tar zxvf inotify-tools-3.14.tar.gz -C /opt

cd /opt/inotify-tools-3.14/
./configure
make && make install

perform inotifywait Command to monitor

1.cd /opt
echo '111' >>1.txt   # Add files to the directory 、 Moving files 
2. Open another terminal （ Or the initiator ） perform inotifywait Command to monitor , Tracking screen output 
inotifywait -mrq -e modify,create,move,delete /opt

Options	explain
-e	Specify the events to monitor
-m	Continuous monitoring
-r	Recursive entire directory
-q	Simplify the output

Writing trigger scripts （ client ）

vim jiankong.sh

#!/bin/bash
INOTIFY_CMD="inotifywait -mrq -e create,delete,move,modify,attrib /var/www/html/"
RSYNC_CMD="rsync -apzH --delete --password-file=/etc/1.pass /var/www/html/ [email protected]::wwwroot/"
$INOTIFY_CMD | while read DIRECTORY EVENT FILE
do
    if [ $(pgrep rsync | wc -l) -le 0 ] ; then
        $RSYNC_CMD
	fi
done

chmod +x jiankong.sh
mkdir -p /var/www/html/
chmod 777 /var/www/html/     
ls -ld /var/www/html/  
chmod +x /etc/rc.d/rc.local
echo "/root/jiankong.sh" >> /etc/rc.d/rc.local

View uplink results

 client 
cd /var/www/html
echo 111 >> 2.txt

 Server side 
cd /var/www/html
ls      # Than before 2.txt

Use rsync To quickly delete a large number of files

demand

If you want to be in Linux Delete a large number of files under , such as 100 ten thousand 、1000 ten thousand , image /usr/local/nginx/proxy_ temp Of nginx Cache, etc , that rm -rf * It may not be easy to use , Because it takes a long time . In this case, we can use rsync To deal with .rsync The principle of substitution is actually used

step

Create an empty folder and add files

mkdir /root/blank    # Empty folder 
mkdir /opt/gxd
cd /opt/gxd
touch {1..1000}.txt  # Simulate large cache file 

use rsync Delete target directory

rsync --delete-before -a -H -v --progress --stats /root/blank/ /opt/gxd               # So the target directory is quickly cleared 

ls /opt/gxd