当前位置：网站首页>Shell script topic (07): file from cfs to bos

Shell script topic (07): file from cfs to bos

2022-08-01 18:53:00 【51CTO】

一、背景

Production produces thousands of audio files every day,There are several sizesG.这些文件存放在cfs内,由于cfsThe usage fee is higher,容量较小,Periodic manual transfers to larger spaces are required for less costbos内.This procedure is to change the manual operation to the automatic way.Involves log output、文件比对、文件复制、文件删除、监控告警等.

二、名词解释

cfs：文件储存CFSare file and protocol storage,It can be understood as a higher-level object storageBOS,文件储存CFSCan't do calculations,Just gateway protocol storage,Reading and writing of pictures and documents、and higher level of security.

bos：对象储存BOSSimilar to Baidu Cloud Disk,But it's enterprise level,对象储存BOS提供一个简单的 Web 服务接口,Can be used to store and extract any amount of data.

cfsShared use by multiple servers,Use cost ratiobos高很多.

三、Program working logic

Most of the audio files generated each day are stored in date-named directories,This directory also has one level of subdirectoriesxxdata,A small number of recording files are stored in this subdirectory.首先将n天前的cfsDate directory copied tobos,Then compare whether they are consistent,Alignment containsmd5和diffand file count comparison,If they are consistent, delete the originalcfsRecording file directory,Exit if there is an inconsistency,Finally, a successful notification or a failed alarm is sent through WeChat.

四、日志说明

Every step of the program has a detailed log record,Including the existence of the source file directory and the target directory、Which files were copied、删除了哪些文件、md5Generation and alignment、Which operations failed etc.The normal operation of the program or the failure of the operation can be viewed through the log location.

五、各模块介绍

本程序共有9个模块,下面一一介绍

1.日志模块logs_bak

logs_bak(){
cp ${LOCAL_LOG}/log.txt ${LOCAL_LOG}/log_${DATE}.txt
echo '' > ${LOCAL_LOG}/log.txt
cp ${ALTER_LOG}/alter.txt ${ALTER_LOG}/alter_${DATE}.txt
echo  -n "" > ${ALTER_LOG}/alter.txt
cp ${MD5_LOG}/md5.txt ${MD5_LOG}/md5_${DATE}.txt
echo '' > ${MD5_LOG}/md5.txt
}

     1.
2.
3.
4.
5.
6.
7.
8.

shell脚本专题(07)：文件由cfs到bos_cfs

本程序共有4个日志,分别是all：Record all actions during the execution of this program;alter：Only the error content of each step is recorded,The log is empty if successful;cron：Record the execution of scheduled tasks every day;md5：记录md5Generation and alignment.

除了cron日志外,The other three logs will be backed up by date cutover when the program is running.

2.Directory inspection modulefile_check

file_check(){
if [ -d ${SOURCE_PATH} ]
then
    echo "the org_directory_file is existed,now you can cp files from ${SOURCE_PATH}/${DATE}" >> ${LOCAL_LOG}/log.txt 
else
    echo "there is no $DATE directory,please check" | tee -a  ${LOCAL_LOG}/log.txt  ${ALTER_LOG}/alter.txt
    exit 101
fi
if [ -d ${DEST_PATH} ]
then
    echo "the dest_directory is existed,now you can cp files to ${DEST_PATH}" >> ${LOCAL_LOG}/log.txt
else
    echo "there is no bos directory,please check" | tee -a  ${LOCAL_LOG}/log.txt  ${ALTER_LOG}/alter.txt   
    exit 102
fi
}

     1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.

Check the source file directorySOURCE_PATHand backup directoryDEST_PATH是否存在,因为有可能出现cfs或者bosMount failed problem.Continue if all paths exist, otherwise report an error and exit.

运行日志截图：

shell脚本专题(07)：文件由cfs到bos_自动化_02

3.生成源文件md5value modulemd5_a

md5_a(){
for  list_a in `find ${SOURCE_PATH}/* \( -path $NODATA \) -prune -o -type f -print |grep -v md5` 
  do
    FILE_NAME_A=${list_a##*/}
    cd ${SOURCE_PATH}
    md5sum ${FILE_NAME_A} >> ${SOURCE_PATH}/md5_a.txt
  done
if [ "$?" -eq "0" ]
then
	echo "the first md5 file md5_a.txt is created" | tee -a  ${LOCAL_LOG}/log.txt ${MD5_LOG}/md5.txt
else
	echo "the first md5 file md5_a.txt created fail,please check" | tee -a  ${LOCAL_LOG}/log.txt  ${ALTER_LOG}/alter.txt
	exit 103
fi
}

     1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.

Get a list of audio files in the first level of the date directory and generate themmd5文件md5_a.txt,The file is being copied tobos后进行md5will be used for comparison.Continue if the file is generated normally,Otherwise, write to the alarm log and exit.

运行日志截图：

shell脚本专题(07)：文件由cfs到bos_md5_03

shell脚本专题(07)：文件由cfs到bos_自动化_04

4.生成源文件md5value modulemd5_b

md5_b(){
for  list_b in `find ${NODATA} -type f -print | grep -v md5` 
  do
    cd ${NODATA}
    FILE_NAME_B=${list_b##*/}
    md5sum ${FILE_NAME_B} >> ${NODATA}/md5_b.txt
  done
if [ "$?" -eq "0" ]
then
	echo "the second md5 file md5_b.txt is created" | tee -a  ${LOCAL_LOG}/log.txt ${MD5_LOG}/md5.txt
else
	echo "the second md5 file md5_b.txt created fail,please check" | tee -a  ${LOCAL_LOG}/log.txt  ${ALTER_LOG}/alter.txt
	exit 104
fi
}

     1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.

Get the list of recording files in the second level of the date directory and generate themmd5文件md5_b.txt,The file is being copied tobos后进行md5will be used for comparison.Continue if the file is generated normally,Otherwise, write to the alarm log and exit.

运行日志截图：

shell脚本专题(07)：文件由cfs到bos_自动化_05

shell脚本专题(07)：文件由cfs到bos_bash_06

5.File copy modulefile_cp

file_cp(){
echo "now ready to copy files from source to dest" | tee -a ${LOCAL_LOG}/log.txt
cp -R ${SOURCE_PATH}  ${DEST_PATH} -vvv >> ${LOCAL_LOG}/log.txt
if [ "$?" -eq "0" ]
then
	echo "the files copy sucessed" | tee -a  ${LOCAL_LOG}/log.txt
else
	echo "the files copy failed,please check" | tee -a  ${LOCAL_LOG}/log.txt  ${ALTER_LOG}/alter.txt
	exit 105
fi
}

     1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.

通过cpThe command will record the file bycfs复制到bos,A detailed log is recorded during the replication process.Continue if the copy is successful,Otherwise, write to the alarm log and exit.

运行日志截图：
shell脚本专题(07)：文件由cfs到bos_cfs_07

shell脚本专题(07)：文件由cfs到bos_cfs_08

6.md5比较模块diff_md5_a

diff_md5_a(){
cd ${DEST_PATH}/${DATE}
echo "this is md5_a compared log"  >> ${MD5_LOG}/md5.txt
md5sum -c md5_a.txt >> ${MD5_LOG}/md5.txt
if [ "$?" -eq "0" ]
then
    echo "the files in ${DATE} except nodata compared Ok" >> ${LOCAL_LOG}/log.txt
	echo "md5_a Ok"
else
    echo "the files in ${DATE} except nodata compared error,please check" | tee -a  ${LOCAL_LOG}/log.txt  ${ALTER_LOG}/alter.txt 
    exit 106
fi
}

     1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.

在bosThe first level target directory to run’md5sum -c md5_a.txt’Make the recording file of the first layermd5比对,If the comparison is normal, continue,Otherwise, write to the alarm log and exit.

运行日志截图：

shell脚本专题(07)：文件由cfs到bos_cfs_09

7.md5比较模块diff_md5_b

diff_md5_b(){    #Compare the second-level files of the target directory
cd ${DEST_PATH}/${DATE}/nodata
echo "this is md5_b compared log"  >> ${MD5_LOG}/md5.txt
md5sum -c md5_b.txt >> ${MD5_LOG}/md5.txt  
if [ "$?" -eq "0" ]
then
    echo "the files nodata compared Ok" >> ${LOCAL_LOG}/log.txt
	echo "md5_b Ok"
else
    echo "the files nodata compared error,please check" | tee -a  ${LOCAL_LOG}/log.txt  ${ALTER_LOG}/alter.txt
    exit 107
fi
}

     1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.

在bosrun from the second level target directory’md5sum -c md5_b.txt’Make the recording file of the second layermd5比对,If the comparison is normal, continue,Otherwise, write to the alarm log and exit.

运行日志截图：

shell脚本专题(07)：文件由cfs到bos_bos_10

8.Source file remove moduleorg_file_rm

org_file_rm(){
diff ${SOURCE_PATH}  ${DEST_PATH}/${DATE}
result1=$?
diff ${NODATA}  ${DEST_PATH}/${DATE}/nodata
result2=$?
wc1=`ls -lR ${SOURCE_PATH} | grep "^-" | wc -l`
wc2=`ls -lR ${DEST_PATH}/${DATE} | grep "^-" | wc -l`
if [ ${result1} -eq "0" ] && [ ${result2} -eq "0" ] &&  [ ${wc1} -eq ${wc2} ] 
then
    echo "the files diff Ok, you can rm the source files" >> ${LOCAL_LOG}/log.txt
    rm -rfv ${SOURCE_PATH} >> ${LOCAL_LOG}/log.txt
else 
    echo "the files diff error,please check" | tee -a  ${LOCAL_LOG}/log.txt  ${ALTER_LOG}/alter.txt
    exit 108
fi
}

     1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.

The first layer and the second layer will be performed before the automatic deletion of the source recording filediffAlignment and alignment of the number of source and target files,Continue if all three conditions are met,If one is not satisfied, write the alarm log and exit.

运行日志截图：

shell脚本专题(07)：文件由cfs到bos_bos_11

shell脚本专题(07)：文件由cfs到bos_自动化_12

9.Alarm or notification module

#!/bin/bash
DATE=`date -d "20 day ago" +%Y-%m-%d`
cd /log/cfs2bos/alter
if [ -s alter.txt ]; then
    cat alter.txt | while read line;do echo $line;done
    exit 1
else
    echo "the files ${DATE} copy sucessed"
    exit 2
fi

     1.
2.
3.
4.
5.
6.
7.
8.
9.
10.