Rsync remote synchronization (sunset is especially gentle, and the world is romantic)

One 、rsync Introduce

1.1 rsync brief introduction

rsync（Remote Sync, Remote synchronization ） Is an open source fast backup tool , It can mirror and synchronize the whole directory tree between different hosts , Support incremental backup , And keep links and permissions , And the optimized synchronization algorithm is used , Perform compression before transmission , So it is very suitable for remote backup 、 Image server and other applications .

rsync The website of our official website is http://rsync.samba.org/, The latest version is 3.1.3, from Wayne Davison For maintenance . As one of the most commonly used file backup tools ,rsync Tend to be Linux and UNIX System silence Recognize one of the basic components installed .

1.2 rsync characteristic

• Support for copying special files , Such as connection file 、 Equipment etc. .
• It can exclude the synchronization of specified files or directories , It's like a pack order tar The exclusion function of .
• Can do to maintain the original file or directory permissions 、 Time 、 Hard and soft links 、 Belong to 、 All attributes such as group do not change –p.
• Incremental synchronization can be achieved , It only synchronizes the changed data , So data transmission efficiency is very high （tar-N）.
• have access to rcp、rsh、ssh And so on （rsync It doesn't encrypt data itself ）.
• Can pass socket（ Process mode ） Transfer files and data （ Server and client ）*****.
• Support anonymous live Authentication （ No need for system users ） Process mode transfer of , Can achieve convenient and safe data backup and mirror .

Two 、rsync Sync source server

In the remote synchronization task , Responsible for initiating rsync The client of the synchronous operation is called the initiator , And is responsible for responding to requests from clients rsync The server that synchronizes is called the synchronization source .

• Synchronize in the downlink （ download ） in , The synchronization source is responsible for providing the original location of the document , The initiator should have read permission to this location
• Synchronize on the uplink （ Upload ） in , The synchronization source is responsible for providing the target location of the document , The initiator should have write permission to this location

3、 ... and 、 To configure rsync Downlink synchronization

rsync The source server 192.168.109.131

# Turn off firewall 
systemctl stop firewalld
systemctl disable firewalld
setenforce 0
# The general system has been installed by default rsync
rpm -q rsync							

# establish /etc/rsyncd.conf  The configuration file 
vim /etc/rsyncd.conf				# Add the following configuration items 

uid = root    # Not in production root user 
gid = root
use chroot = yes					# Locked in the source directory 
address = 192.168.109.131				# Monitor address 
port = 873			# Listening port  tcp/udp 873, It can be done by cat /etc/services | grep rsync see 
log file = /var/log/rsyncd.log				# Log file location 
pid file = /var/run/rsyncd.pid				# Store process  ID  File location of 
hosts allow = 192.168.109.0/24				# The client address allowed to access 
dont compress = *.gz *.bz2 *.tgz *.zip *.rar *.z		# File types that are no longer compressed during synchronization 

[wwwroot]					   # Shared module name 
path = /var/www/html			# The actual path to the source directory 
comment = folder of web          # Notes 
read only = yes					# Is it read-only 
auth users = backuper			# Authorized account , Multiple accounts are separated by spaces 
secrets file = /etc/rsyncd_users.db		# Data file for storing account information 
# Such as anonymity , Just put one of them “auth users” and “secrets file” The configuration item can be removed .

# Create data file for backup account 
vim /etc/rsyncd_users.db
backuper:000000				# There is no need to create a system user with the same name 

# Grant authority 
chmod 600 /etc/rsyncd_users.db

# Ensure that all users have access to the source directory /var/www/html  Have read access 
mkdir -p /var/www/html
chmod +r /var/www/html/
ls -ld /var/www/html/

# Write some shared files 
[[email protected] ~]# cd /var/www/html/
[[email protected] html]# ls
[[email protected] html]# echo a > a
[[email protected] html]# echo b > b
[[email protected] html]# echo c > c
[[email protected] html]# mkdir abc
[[email protected] html]# echo 'this is stevelu`s test file' >abc/index.html
[[email protected] html]# ls
a  abc  b  c
[[email protected] html]# 


# start-up  rsync  Service program 
rsync --daemon			# start-up  rsync  service , In the way of independent monitoring service （ Daemon ） function  
# View port number 
netstat -anpt | grep rsync

# close  rsync  service 
kill $(cat /var/run/rsyncd.pid)
rm -rf /var/run/rsyncd.pid

# You can also write your own script to manage the service 

Four 、 Initiator configuration

client 192.168.109.132

# Turn off firewall 
systemctl stop firewalld
systemctl disable firewalld
setenforce 0

4.1 The basic format

rsync [ Options ] Original location Target location

Most backup programs require that the original location be specified 、 Target location ,rsync It's the same with orders . The simplest rsync The usage is similar to cp command .

for example , You can /etc/fstab、 Catalog /boot/grub Synchronous backup to /opt Under the table of contents , among “-r” Option means recursion of the entire directory tree ,“-l” Option to back up linked files .

rsync -rl /etc/fstab /boot/grub /opt 

4.2 Common options

Options	meaning
-r	Recursive mode , Contains all the files in the directory and subdirectories
-l	For symbolic link files, still copy as symbolic link files
-v	Show details of the synchronization process （verbose） Information
-z	Compress when transferring files （compress）
-a	Archiving mode , Keep file permissions 、 Properties and other information , Equivalent to combination options “-rlptgoD”
-p	Keep the permission tag of the file
-t	Keep the time stamp of the file
-g	Keep the group tag of the file （ For super users only ）
-o	Keep the owner tag of the file （ For super users only ）
-H	Keep hard connection files
-A	Retain ACL Attribute information
-D	Keep equipment files and other special files
–delete	Delete files that exist in the target location but not in the original location
–checksum	According to the check sum （ Not the file size 、 Modification time ） To decide whether to skip the file

When deleting thousands of files

• rm Delete slower
• rsync --delete Delete fast

4.3 There are two ways to download to local

Download the specified resource to the local /data Backup under Directory

4.3.1 Method 1

# user name @ The host address : : Share module name 
rsync -avz [email protected]::wwwroot /data/   # password 000000

#backuper It refers to which user identity I use when synchronizing 
#wwwroot It's a module , The default path of synchronization and some features will be written below the module , So we just need to write modules 
#/opt It refers to the synchronization to the local directory 

4.3.2 Method 2

#rsync:/ user name @ The host address / Share module name 
rsync -avz rsync://[email protected]/wwwroot /test/

#URL： Specific location point , for example ：http://www.stevelu.com/index.html

4.3.3 Planned tasks

In order not to enter a password during synchronization , Need to create a password file , preservation backuper User's password , Such as /etc/server.pass. In execution rsync Use options when synchronizing “–password-file=/etc/server.pass” Just specify

# No interactive format configuration 
# Create password file 
echo 000000 > /root/rsync_password
# Set the permissions 
chmod 600 !$
# Create planned tasks 
crontab -e
# Every two hours 
* */2 * * * /usr/bin/rsync -avz --password-file=/root/rsync_password [email protected]::wwwroot /data

systemctl restart crond
systemctl enable crond

5、 ... and 、reync Real time synchronization （ The upside ）

5.1 principle

• The lack of regular synchronization

1. Perform backups at a fixed time , The delay is obvious 、 Poor real-time performance
2. When the synchronous source does not change for a long time , Intensive regular tasks are unnecessary

• The advantages of real-time synchronization

1. Once the synchronization source changes , Start backup now
2. As long as the synchronization source does not change , Do not perform backup

Linux Kernel from 2.6.13 The version begins to provide inotify Notification interface , Used to monitor changes in the file system , Such as file access 、 Delete 、 Move 、 Modify etc. . Using this mechanism , It is very convenient to realize file change alarm 、 Incremental backup , And respond to changes in directories or files in a timely manner .

take inotify Mechanism and rsync Combination of tools , You can achieve triggered backup （ Real time synchronization ）—— As long as the original bit The set document changes , Start the incremental backup operation immediately ; Otherwise, it will be in a state of silent waiting , As shown in the figure below . such , This avoids the latency when backing up according to a fixed cycle 、 The period is too dense and so on

because inotify The notification mechanism consists of Linux Provided by kernel , Therefore, it is mainly used for local monitoring , It is more suitable for uplink synchronization when applied in triggered backup .

5.2 Initiator configuration rsync+inotify

5.2.1 modify rsync Source server configuration file

vim /etc/rsyncd.conf
......
read only = no		# Turn off read-only , Uplink synchronization needs to be able to write 

# restart rsync
kill $(cat /var/run/rsyncd.pid)
# If the service fails to start, an error will be reported , Will serve pid File deletion 
rm -rf /var/run/rsyncd.pid
rsync --daemon	
netstat -anpt | grep rsync

# Give write permission 
mkdir /data
chmod 777 /data/

5.2.2 Adjust the client inotify Kernel parameters

stay Linux The kernel , default inotify The mechanism provides three regulatory parameters ：max_queue_events（ Monitor the event queue , The default value is 16384）、max_user_instances（ The maximum number of monitoring instances , The default value is 128）、max_user_watches（ The maximum number of monitoring files per instance , The default value is 8192）. When you want to monitor the directory 、 When the number of files is large or changes frequently , It is suggested to increase the values of these three parameters .

cat /proc/sys/fs/inotify/max_queued_events
cat /proc/sys/fs/inotify/max_user_instances
cat /proc/sys/fs/inotify/max_user_watches

# modify 
vim /etc/sysctl.conf
fs.inotify.max_queued_events = 16384
fs.inotify.max_user_instances = 1024
fs.inotify.max_user_watches = 1048576

sysctl -p

5.2.3 Client installation inotify-tools

use inotify The mechanism also needs to be installed inotify-tools, To provide inotifywait、inotifywatch Auxiliary tool program , Used for monitoring 、 Summarize the changes .

inotifywait： Can be monitored modify（ modify ）、create（ establish ）、move（ Move ）、delete（ Delete ）、attrib（ Property changes ） And so on , Output results as soon as there is a change .

inotifywatch： It can be used to collect file system changes , At the end of the run, output the change of the summary .

# Upload and unzip 
cd /opt/
tar zxvf inotify-tools-3.14.tar.gz -C /opt/

cd /opt/inotify-tools-3.14
# Compilation and installation 
./configure
make && make install

# You can do it first “inotifywait” command , Then open a new terminal to  /var/www/html  Add files to directory 、 Moving files , Track the screen output results in the original terminal .
inotifywait -mrq -e modify,create,move,delete /data

1. Options “-e”： Used to specify which events to monitor
2. Options “-m”： Continuous monitoring
3. Options “-r”： Represents recursion of the entire directory
4. Options “-q”： Simplify the output

5.2.4 Write triggered synchronization scripts

Write trigger synchronization script at another terminal （ Be careful , Script name cannot contain rsync character string , Otherwise the script may not take effect ）

vim /opt/inotify.sh
#!/bin/bash

# Definition inotifywait monitor /data Variables for file events in the directory 
INOTIFY_CMD="inotifywait -mrq -e modify,create,attrib,move,delete /data"

# Define execution rsync Uplink synchronization variables 
RSYNC_CMD="rsync -azH --delete --password-file=/root/rsync_password /data [email protected]::backupdir"
# Use while、read Continuously obtain monitoring results , According to the result, it can be further judged whether the output monitoring record is read 
$INOTIFY_CMD | while read DIRECTORY EVENT FILE
do
    if [ $(pgrep rsync | wc -l) -le 0 ] ; then
	# If rsync Not executing , Start immediately 
        $RSYNC_CMD
    fi
done


chmod +x /opt/inotify.sh
chmod 777 /data
chmod +x /etc/rc.d/rc.local
echo '/opt/inotify.sh' >> /etc/rc.d/rc.local				# Add boot auto execution 

The above script is used to detect native /data Changes in the catalogue , Once there is an update trigger rsync Synchronous operation , Upload backup to server 192.168.109.131 Of /data Share Directory .

The verification process of triggered uplink synchronization is as follows ：
（1） Running on local machine /opt/inotify.sh Script program .
（2） Switch to the local /data Catalog , Execution increase 、 Delete 、 Modify files and other operations .
（3） View the in the remote server /data Changes in the directory .

6、 ... and 、 Use rsync To quickly delete a large number of files

If you want to be in linux Delete a large number of files under , such as 100 ten thousand 、1000 ten thousand , image /usr/local/nginx/proxy_temp Of nginx Cache, etc , that rm -rf * It may not be easy to use , Because it takes a long time . In this case, we can use rsync To deal with .rsync The principle of substitution is actually used .

First create an empty folder ：

mkdir /home/blank

use rsync Delete target directory ：

rsync --delete-before -a -H -v --progress --stats /home/blank/ /usr/local/nginx/proxy_temp

So the target directory is quickly cleared

Option description

Options	explain
–delete-before	The receiver deletes during transmission
-a	Archiving mode , Indicates that files are transferred recursively , And keep all file properties
-H	Keep hard connected files
-v	Detailed output mode
–progress	Display the transmission process during transmission
–stats	Give the transfer status of some files