Shell timing script, starting from 0, CSV format data is regularly imported into PostgreSQL database shell script example

Preface ：
As shown in the title , This article is from 0 Start , For beginners .
Students with a certain foundation , You can jump to the part you need , Choose to watch .
All commands and scripts involved in this article , Have been tested by the author .
If there is a problem or clerical error , Welcome to communicate and correct .

One 、shell Script related knowledge and the first shell Script example writing ;
No contact with shell Script friends , Let's write a shell An example of a script ;
1. So let's set up a shell Script ;
stay linux On vim demoShell.sh

#/bin/bash
# shell In the syntax # It's an annotation , above #/bin/bash Is the script startup item , commonly shell Scripts start with this ;

# echo It's output ,sh When executing a script , It will show the contents in double quotation marks .
#  It can also be done through >>( Do not overwrite the previous content ) perhaps >( Overwrite previous content ) It will be implemented echo The content is written to the specified place to form a log ;
echo "demoShell start."
# [`date +%y%m%d/%H:%M:%S`] Is to obtain the current time of the system , Specific date , A function of hours, minutes and seconds .
# This output is for later scheduled tasks , Verify whether the scheduled task is effective ;
echo "[`date +%y%m%d/%H:%M:%S`]"
echo "demoShell finish." 

It can be executed sh demoShell.sh,linux The output will be displayed on 3 individual echo Content ;
2. Write another startup demoShell.sh Start script for runDemoShell.sh, With the above >> Method , Will perform demoShell.sh Record the situation of to the path of the corresponding folder , Convenient for troubleshooting and positioning in case of errors ;

#/bin/bash
# 1>> It means normal log of normal operation input ,2>> yes error Log input to error journal ;
sh /home/admin/demoShell.sh 1>>/home/admin/copyDataShellLog/copyDataShellLog.log 2>>/home/admin/copyDataShellLog_err/copyDataShellLog_err.log

Two 、 use corntab Set up scheduled tasks ;
First of all, let's get to know corntab Relevant knowledge and common commands ;
crontab Specify the executing user ：
You can enter the configuration file . Configure at this time crontab The executor of is the currently logged in user ,
If the current user is root, It needs to be configured for other users , have access to crontab -e -u user name .

// View user commands ;

crontab -l

// Set timing script ;

crontab -e

// Execute every minute ;

*/1 * * * * sh /home/admin/runDemoShell.sh

Example ：
Every time 10min Perform a task ;

*/10 * * * * /scripts/monitor.sh

Perform a task every four hours ;

0 */4 * * * /scripts/script.sh

Common commands ：
// Overload configuration ;

/sbin/service crond reload

// Timing start ;

/sbin/service crond start

// Timed off ;

/sbin/service crond stop

// Restart regularly ;

/sbin/service crond restart

// Check the timing status ;

/sbin/service crond status

Then follow the , Establish a scheduled task step to set a scheduled task , And inspection .
1. Use crontab -e Under the required user permissions , Create a scheduled task ;
2./sbin/service crond reload Overload configuration ;/sbin/service crond restart Restart regularly ;/sbin/service crond status Check the timing status ;
For example, we write a scheduled task that is executed every minute
// Execute every minute ;

*/1 * * * * sh /home/admin/runDemoShell.sh

And then you can do it in /home/admin/copyDataShellLog/copyDataShellLog.log In the log , See the scheduled task record that runs every minute ;

3、 ... and 、csv Format data is imported regularly psql database shell Script example ;
After understanding shell Scripts and crontab After that , Let's write practical examples ;
Example needs ：
In a data folder /data/sftpuser There is 8 A compressed package of data with different names , for example 4G_USER_PERCEPTION_H2021102012.tar.
1. Automatic execution with scheduled tasks , And make log records ;
2. According to the name and day granularity , Hourly granularity , take tar According to the name and day granularity , Hourly granularity , take tar( perhaps lzo) Format pack , Unzip to the specified file /data/csv/ in ;
3. The specified folder /data/csv/4G_CELL_PERCEPTION_H, Unzip it. Okay csv file , Import psql database ( Import psql database , Must be antdb User permissions , To operate . Specific examples are as follows );
4. After importing the table , Backup /data/sftpuser File to new folder , And delete /data/csv/4G_CELL_PERCEPTION_H Unzipped file ;

Script example ：

#!/bin/bash
. /etc/profile
. ~/.bash_profile

yesterday=`date -d last-day +%Y%m%d`
echo $yesterday

# user Yes, it is lzo Compress , You need to perform decompression lzo Corresponding command ,lzop -d $file,  still more mv become csv file 
#  Get the specified file /data/sftpuser/day/4G_USER_PERCEPTION_20211021/000299_0/lzo, Unzip to the specified folder /data/4g/hour/4G_USER_PERCEPTION_D
echo " Copy to the extraction folder ,cp start.[`date +%y%m%d/%H:%M:%S`]"

if [ ! -d "/data/4g/day/ti/4G_USER_PERCEPTION_D/`date +%y%m%d`" ] then mkdir /data/4g/day/ti/4G_USER_PERCEPTION_D/`date +%y%m%d` fi if [ ! -d "/data/bak/4g/day/ti/4G_USER_PERCEPTION_D/`date +%y%m%d`" ] then mkdir /data/bak/4g/day/ti/4G_USER_PERCEPTION_D/`date +%y%m%d` fi cd /data/sftpuser/day/ function read_dir(){ for file in `ls $1` # Notice here that these are two backquotes , Indicates running the system command  do if [ -d $1"/"$file ] # Note that there must be a space between them , Otherwise, an error will be reported  then read_dir $1"/"$file else echo $1"/"$file # Just process the file here  cp $1"/"$file /data/4g/day/ti/4G_USER_PERCEPTION_D/`date +%y%m%d`/$file fi done } # Read the first parameter  read_dir /data/sftpuser/day/4G_USER_PERCEPTION_$yesterday echo " Copy complete ,cp finish.[`date +%y%m%d/%H:%M:%S`]" ##  Unzip in the unzip folder  cd /data/4g/day/ti/4G_USER_PERCEPTION_D/`date +%y%m%d` echo " Unzip the file ,lzop start.[`date +%y%m%d/%H:%M:%S`]" for file in $(ls *|grep -E ".lzo") do echo $file lzop -d $file; #  Plus, after unzipping, move the file to bak Operation of backing up files , So as not to repeat it next time ; Untie the notes after the test is ok ; mv /data/4g/day/ti/4G_USER_PERCEPTION_D/`date +%y%m%d`/$file /data/bak/4g/day/ti/4G_USER_PERCEPTION_D/`date +%y%m%d`/$file; done; echo " The file is unzipped ,lzop finish.[`date +%y%m%d/%H:%M:%S`]" ###  Rename the extracted file ; cd /data/4g/day/ti/4G_USER_PERCEPTION_D/`date +%y%m%d` echo " rename ,mv start.[`date +%y%m%d/%H:%M:%S`]" for file in $(ls *|grep -E "_[0-9]{
    1}") do echo $file mv /data/4g/day/ti/4G_USER_PERCEPTION_D/`date +%y%m%d`/$file /data/4g/day/ti/4G_USER_PERCEPTION_D/`date +%y%m%d`/$file.csv done; echo " Rename complete ,mv finish.[`date +%y%m%d/%H:%M:%S`]" ###  The specified folder /data/4g/day/ti/ Of csv file ,copy To psql cd /data/4g/day/ti/4G_USER_PERCEPTION_D/`date +%y%m%d` echo " Will specify the folder's csv file ,copy To psql Designated table ,copy start.[`date +%y%m%d/%H:%M:%S`]" #SET search_path TO aioss It's choice SCHEMA; Create table , And circular insertion should be separated . Otherwise an error , hinder sql It will not be implemented ; psql -d aioss -U antdb -c "SET search_path TO aioss; CREATE TABLE ti_qm_lte_cell_user_kpi_d_$yesterday PARTITION OF ti_qm_lte_cell_user_kpi_d FOR VALUES IN ($yesterday); " for file in $(ls * |grep -E ".csv") do echo $file # ALTER TABLE ti_qm_lte_cell_user_kpi_d DROP PARTITION 'ti_qm_lte_cell_user_kpi_d_$yesterday'; psql -d aioss -U antdb -c "SET search_path TO aioss; CREATE TABLE ti_qm_lte_cell_user_kpi_d_$yesterday PARTITION OF ti_qm_lte_cell_user_kpi_d FOR VALUES IN ($yesterday); copy ti_qm_lte_cell_user_kpi_d FROM '/data/4g/day/ti/4G_USER_PERCEPTION_D/`date +%y%m%d`/$file' delimiter '|' csv header;" done; echo " Data import complete ,copy finish.[`date +%y%m%d/%H:%M:%S`]" ###  Copy and compress , Backup /data/sftpuser Data to /home/antdb/bak/ # PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:~/bin # export PATH # bakdir="/home/antdb/bak/" # filename="`date +%y%m%d`_dataBak.tar.gz" # echo " Start backup ,begin bak " # if [ ! -x "$bakdir" ];then # mkdir $bakdir # fi # cd $bakdir # tar cvfz $filename /data/sftpuser # echo " Backup complete ,bak finish. " ###  Copy and compress , Backup /home/antdb/testForTar/4G_CELL_PERCEPTION_H Data to /home/antdb/bak/ # PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:~/bin # export PATH # bakdir="/home/antdb/bak/" # filename="`date +%y%m%d`_dataBak.tar.gz" # echo " Start backup ,begin bak " # if [ ! -x "$bakdir" ];then # mkdir $bakdir # fi # cd $bakdir # tar cvfz $filename /home/antdb/testForTar/4G_CELL_PERCEPTION_H # echo " Backup complete ,bak finish. "

Be careful
1. We must add

. /etc/profile
. ~/.bash_profile

Otherwise, an error will appear during operation ;
2. Create table , And circular insertion should be separated . Otherwise an error , hinder sql It will not be implemented ;

Regularly delete scripts ：

#!/bin/bash
. /etc/profile
. ~/.bash_profile

# The current day 
today=`date +%Y%m%d`
echo $today

#30 Days ago, 
day30ago=`date -d "30 days ago" +%Y%m%d`
echo $day30ago

# user Table directory /data/sftpuser/day
#  Read the file /data/sftpuser/day/4G_USER_PERCEPTION_20211114/
cd /data/sftpuser/day/
function read_dir(){
    
for file in `ls $1` # Notice here that these are two backquotes , Indicates running the system command 
do
 if [ -d $1"/"$file ] # Note that there must be a space between them , Otherwise, an error will be reported 
 then
 read_dir $1"/"$file
 else
 echo $1"/"$file # Just process the file here 
 #  Delete file 
 #rm -rf $1"/"$file /data/sftpuser/day/$day30ago
 #  test 
 rm -rf /home/antdb/day/4G_USER_PERCEPTION_$day30ago
 fi
done
} 
# Read the first parameter 
read_dir /home/antdb/day/4G_USER_PERCEPTION_$day30ago

echo " Delete file completed ,`date -d "30 days ago" +%Y%m%d`,`date +%Y%m%d`"

Four 、psql Relevant operations and check data quality ;
Yes antdb User's folder , To open the corresponding permission , And adjusted to antdb grouping ;
chmod -R 777 Folder name
chown antdb:antdb Folder name

stay antdb Under the user , Input psql Access to psql database ;
\q Is out of ;
Use \l Used to view existing databases ：
Next use \c + Database name To access the database ：
\d tablename Look at the table information ：
Then you can go through sql Make inquiries such as ：select * from Table name where Conditions limit 3;
( Pay attention to marking ";", Otherwise, it won't execute ; And try to hit limit x, Otherwise big data , Too much presentation data , Create a Caton ;)

establish SCHEMA The order is :
CREATE SCHEMA aioss;
\dn – see schemas
psql Middle switch schema:
perform ：set search_path to aioss;
The subsequent operations are all aimed at test_schema This schema 了 ;