Rsync remote synchronization

rsync brief introduction ：

rsync It's an excellent 、 fast 、 Multi platform local or remote data image synchronous backup tool . Apply to Unix/Linux/Windows And so on .

When synchronizing backups , By default ,rsync Through its unique quick check Algorithm , Only synchronize files or directories whose size or last modification time has changed （ Or according to the permission , The change of dominant class is synchronous , Parameters need to be specified ） Even synchronize only the changed parts of a file , therefore , It can realize the function of quickly synchronizing data .

rsync and cp,scp The difference between ：

cp,scp The tool copy is a full copy every time , and rsync In addition to full backups , It also has the function of incremental copy , therefore , Better performance and efficiency .

rsync Operation mode ：      rsync There are roughly three main ways to transmit data ：   1. Local data transmission   2. adopt rcp,ssh Wait channel transmission   3. Transfer data as daemons

Local data transmission grammar ：   rsync [OPTION] SRC... [DEST] SRC： Source file DEST： Target file option Parameter description ：    # In general use -avz Can -a： Archiving mode , Recurse and keep object properties , Equate to -rlptgoD -r： Recursive mode , Include all files in the directory and subdirectories -l： For symbolic link files, still copy as symbolic link files -p： Keep the permission tag of the file -t： Keep the time stamp of the file -g： Keep the group tag of the file （ For super users only ） -o： Keep the owner tag of the file （ For super users only ） -D： Keep equipment files and other special files -v： Show details of the synchronization process （verbose） Information -z： Compress when transferring files （compress） -H： Keep hard connection files -A： Retain ACL Attribute information --delete： Delete files that exist in the target location but not in the original location --checksum： Decide whether to skip files based on the checksums of objects

demonstration ： take /etc Copy the files in the directory to /tmp Under the table of contents  [[email protected] ~]# rsync -avz -P /etc /tmp/  # The first copy takes a long time , There is also a lot of output information The second copy is very fast , because /etc There is no data update under the directory  [[email protected] ~]# rsync -avz -P /etc /tmp/ building file list ... 2141 files to consider

sent 61983 bytes  received 20 bytes  124006.00 bytes/sec total size is 88435897  speedup is 1426.32 Be careful ：rsync -avz -P /etc/ /tmp/      Synchronous only etc The contents of the catalogue ,etc Itself is not synchronized         rsync -avz -P /etc /tmp/      hold etc and etc All the contents in it have been tested tmp Under the table of contents By remote shell Data transfer ： grammar ： Pull ：rsync [option] [[email protected]]HOST:SRC...  [DEST] push ：rsync [option] SRC... [[email protected]]HOST:DEST [[email protected]]HOST： by Rsync Synchronous remote connection user and host address SRC： As source , That is, the copied partition 、 Files or directories, etc , and HOST Connect them with a colon ; [DEST] Partition for the purpose of 、 Files or directories, etc

demonstration ： Do it in advance ssh passageway Local etc Directory push to tmp Under the table of contents , Because of ssh passageway , So I didn't ask for a password [[email protected] ~]# rsync -avz -P /etc -e 'ssh -p 22' [email protected]:/tmp/

Will be far away etc Backup the directory to the local tmp Under the table of contents [[email protected] ~]# rsync -avz -P -e 'ssh -p 22' [email protected]:/etc /tmp/

The way of daemons ： [[email protected] ~]# vim /etc/rsyncd.conf uid = root gid = root use chroot = no        # Lock in the home directory max connections = 200  # maximum connection timeout = 300          # Timeout time pid file = /var/run/rsyncd.pid lock file = /var/run/rsync.lock log file = /var/log/rsyncd.log [www]                    # modular , You need to use this name when reading data path= /var/www/html/    # Path of remote backup , Grabbing is also from here , If the user uploads uid and gid Users need upload permission , If Du also needs to have read permission ignore errors            # Ignore mistakes read only = false        # If read-only is false, you can upload data , If it is true Then you can't upload data list = false  hosts allow = 192.168.0.0/24 hosts deny = 0.0.0.0/32 auth users = rbackup                # Authenticated users secrets file = /etc/rsync.password  # Store user and password files

Create a file to store authenticated users [[email protected] ~]# vim /etc/rsync.password rbackup:RedHat [[email protected] ~]# chmod 600 /etc/rsync.password  # The authority must be 600 [[email protected] ~]# ls /etc/rsync.password -l -rw------- 1 root root 15 12-10 10:18 /etc/rsync.password

Start the service [[email protected] ~]# rsync --daemon    #--daemon Means to start as a daemon [[email protected] ~]# ps -ef | grep rsync | grep -v grep root      7915    1  0 10:20 ?        00:00:00 rsync --daemon [[email protected] ~]# netstat -lnt | grep 873 tcp        0      0 0.0.0.0:873                0.0.0.0:*                  LISTEN      tcp        0      0 :::873                      :::*                        LISTEN

Set the power on auto start [[email protected] ~]# echo "/usr/bin/rsync --daemon" >> /etc/rc.d/rc.local [[email protected] ~]# tail -1 /etc/rc.d/rc.local /usr/bin/rsync --daemon

Client Deployment [[email protected] ~]# echo "redhat" > /etc/rsync.password  # Only need password, no account [[email protected] ~]# chmod 600 /etc/rsync.password        # File permissions 600 [[email protected] ~]# cat /etc/rsync.password redhat

Use daemon to transfer data Pull ：rsync[OPTION][[email protected]]HOST::SRC [DEST]       rsync[OPTION] rsync://[[email protected]]HOST[:PORT]/SRC... [DEST] push ：rsync[OPTION] SRC... [[email protected]]HOST::DEST       rsync[OPTION] SRC... rsync://[[email protected]]HOST[:PORT]/DEST The client creates a file and pushes it to The server End [[email protected] html]# touch a.txt [[email protected] html]# rsync -avz -P /var/www/html/ [email protected]::www/ --password-file=/etc/rsync.password perhaps [[email protected] html]# rsync -avz -P /var/www/html/ rsync://[email protected]:/www --password-file=/etc/rsync.password building file list ... 2 files to consider ./ a.txt           0 100%    0.00kB/s    0:00:00 (xfer#1, to-check=0/2)

sent 100 bytes  received 44 bytes  96.00 bytes/sec total size is 0  speedup is 0.00

The server End view push succeeded [[email protected] html]# ls a.txt  index.html  index.php

--deleted Parameters ： Ensure that local data and remote data are completely consistent , There are as many remote places as there are local places , If there is no remote end in the local area, delete Client execution [[email protected] ~]# mkdir aaa Create an empty directory [[email protected] ~]# rsync -avz -P --delete /root/aaa/ [email protected]::www/ --password-file=/etc/rsync.password building file list ... 1 file to consider deleting index.php deleting index.html deleting a.txt ./

sent 50 bytes  received 22 bytes  13.09 bytes/sec total size is 0  speedup is 0.00 Server view /var/www/html There is nothing under the directory [[email protected] ~]# ll /var/www/html/ A total of 0

The server End multiple directories , Configuration file writing [[email protected] ~]# vim /etc/rsyncd.conf uid = root gid = root use chroot = no max connections = 200 timeout = 300 pid file = /var/run/rsyncd.pid lock file = /var/run/rsync.lock log file = /var/log/rsyncd.log ignore errors read only = false list = false hosts allow = 192.168.0.0/24 hosts deny = 0.0.0.0/32 auth users = rbackup secrets file = /etc/rsync.password [www] path = /www [bbs] path = /bbs [blog] path = /blog

After the change, restart the service [[email protected] ~]# pkill rsync && rsync --daemon

rsync+inotify Realize triggered automatic synchronization ,inotify End create or a file rsync The backup source also automatically creates an identical file ,inotify Delete a file rsync The backup source also automatically deletes files .

Be careful ：inotify It's deployed in rsync When the service is successful

demonstration ： Check whether the current system supports inotify [[email protected] ~]# ls -l /proc/sys/fs/inotify/ A total of 0 -rw-r--r-- 1 root root 0 12-11 10:36 max_queued_events  # Monitor the event queue -rw-r--r-- 1 root root 0 12-11 10:36 max_user_instances  # The maximum number of monitoring instances -rw-r--r-- 1 root root 0 12-11 10:36 max_user_watches    # The maximum number of monitoring files per instance

Compilation and installation inotify tar zxf inotify-tools-3.14.tar.gz cd inotify-tools-3.14 ./configure --prefix=/usr/local/inotify-tools-3.14 make && make install

Script

host01=10.0.0.191            # Definition IP src=/data0/www/www/          # Define source directory dst=www                      # Target directory user=rsync_backup            # Backup users rsync_passfile=/etc/rsync.password      # Password file inotify_home=/usr/local/inotify-tools-3.14/  #inotify The catalog of #judge                                        if [ ! -e "$src" ] \ || [ ! -e "${rsync_passfile}" ] \ || [ ! -e "${inotify_home}/bin/inotifywait" ] \ || [ ! -e "/usr/bin/rsync" ]; then   echo "Check File and Folder"              # Judge whether the above file exists. If it does not exist, report an error and exit the execution   exit 9 fi ${inotify_home}/bin/inotifywait -mrq --timefmt '%d/%m/%y %H:%M' --format '%T %w%f' -e close_write,delete,create,attrib $src \ | while read file        # Monitored events delete,create,attrib Pass after monitoring read Read         do     cd $src && rsync -aruz -R --delete ./  --timeout=100 [email protected]$host01::$dst --password-file=${rsync_passfile} >/dev/null 2>&1         done exit 0

Put the script in the background to execute [[email protected] ~]# sh inotify.sh & [1] 8216 [[email protected] ~]# ps -ef | grep inotify root      8216  7503  0 11:15 pts/2    00:00:00 sh inotify.sh root      8217  8216  0 11:15 pts/2    00:00:00 /usr/local/inotify-tools-3.14//bin/inotifywait -mrq --timefmt %d/%m/%y %H:%M --format %T %w%f -e close_write,delete,create,attrib /var/www/html root      8218  8216  0 11:15 pts/2    00:00:00 sh inotify.sh root      8220  7503  0 11:15 pts/2    00:00:00 grep inotify

At this time, create several files [[email protected] ~]# cd /var/www/html/ [[email protected] html]# touch a b c d Backup The server View has been synchronized [[email protected] etc]# cd /var/www/html/ [[email protected] html]# ls a  a.txt  b  c  d