Shell implements basic file operations (SED edit, awk match)

Use Shell Built-in tools , Realize the operation of text files , Meet the needs of daily data processing .

One 、 command sed - Editing tools

sed Is a powerful non interactive streaming text editor , It can replace the words in the file , Process one line at a time , First put the line into the cache , Process in cache , After completion, send the cached content to the terminal .
Basic grammar yes sed [option] [set command ] filename

Option parameters

sed -e Directly in instruction mode sed edit , When there are multiple command lines sed That's what we need -e Options ;
sed -i Modify the content directly , Without this parameter, it is just a preview ;
sed -f The heel is preserved sed Command file ;
sed -n Cancel default output .

set Command function parameters

a add newly added ,a After the connection string ;
c change change , Change the contents of the matching line ;
d delete Delete , Delete matching content ;
i insert Insert , Want to insert content before matching lines ;
s substitute Replace , Replace what matches ;
n Read next line .

demo demonstration

1） Insert data before or after the specified line number
The original file is shown below .

Insert after line , Carry out orders sed -i ‘3a Beijing xicheng ’ info.txt , Add Beijing Xicheng before the third line , Here plus i Parameters , Is to modify the file directly , The results are shown below .

Insert before the line , Carry out orders sed -i ‘3i Beijing changping ’ info.txt , Add Beijing Xicheng before the third line , The results are shown below .

2） Insert content before and after the specified content
Support regular
Insert before the line , Carry out orders sed ‘/ Beijing /ihello’ info.txt , Insert before the beginning of Beijing hello, The results are shown below .

3） Insert content after the last line
Insert after line , Carry out orders sed ‘$alove java’ info.txt , Insert... In the last line love java, The results are shown below .

4） Delete the contents of the specified range
Delete use d Command function , perform sed ‘5,6d’ info.txt, namely Delete 5-6 That's ok , The results are shown below .

Reverse delete , perform sed ‘1,3!d’ info.txt, namely Retain 1-3 That's ok , The results are shown below .

5） Delete the matching line and the following lines
Match the line and its contents , Carry out orders sed ‘/ zhangjiakou /,+1d’ info.txt, Use +1, The execution result is shown in the figure below .

6） Replace the first specified string in the matching line
Match line substitution , Carry out orders sed ‘s/ Beijing / The Beijing municipal /’ info.txt, Use s, Replace Beijing with Beijing , The execution result is shown in the figure below .

Global replacement , Carry out orders sed ‘s/ Beijing / The Beijing municipal /g’ info.txt, Use g The global matching .
Replacement section 2 individual , Carry out orders sed ‘s/ Beijing / The Beijing municipal /2’ info.txt, Appoint 2, It means the first one 2 individual .
Replacement section 2 And write to the file , Carry out orders sed -n ‘s/ Beijing / The Beijing municipal /2pw infodata.txt’ info.txt, Use -n Get matching ,-p Print out ,w write file .
7） Stitching at the end of each line text
Match line substitution , Carry out orders sed ‘s/dollar/& test/’ info.txt, Use s, Replace Beijing with Beijing , The execution result is shown in the figure below .

8） First splicing of each row #
Match line substitution , Carry out orders sed ‘s/^/&#/’ info.txt, Use s, Replace Beijing with Beijing , The execution result is shown in the figure below .

Be careful ： Replace the command of the class , need / To end .

9） Query the matching content
Match the display string , Carry out orders sed -n ‘/ Beijing /p’ info.txt, Use -n Match to ,p Print display , Find Beijing , The execution result is shown in the figure below .

10） Execute multiple sed command
Mode one ： use -e Connect , command sed -e ‘1d’ -e ‘s/ Beijing / The Beijing municipal /g’ info.txt, Multiple sed At the same time , The effect is as follows .

Mode one ： use ; Connect , command sed ‘1d;s/ Beijing / The Beijing municipal /g’ info.txt, Multiple sed At the same time , The effect is as follows .

Two 、 command awk - Analysis tools

awk Powerful text analysis tool , Read the file line by line , Slice each line with a space default separator , The slicing part is subjected to various analytical treatments .
Basic grammar awk option ‘pattern{action}’ filename{}

Option parameters

-F Specify the separator for the input file split ;
-v Assign a user-defined variable .

Built in functions

NF The number of fields in the browsing record , The number of columns divided according to the separator ;
NR Line number , Number of lines read ;
$n $0 Variable value whole record ,$1 The first field in the current line

$NF number finally Represents the last column of information .

demo demonstration

Data preparation ： Copy file cp /etc/passwd ./
1） Match print data
Carry out orders ,echo “ I Love China Beijing ” | awk ‘{print $1"&“$2”&“$3”&"$4}’ , Space segmentation extracts data , Match by placeholder splicing , The execution effect is shown in the figure below .

2） Regular queries
Carry out orders ,ls -a | awk ‘/^c/’ , Print c Opening file , The execution effect is shown in the figure below .

3） Print by column
Print the first column , Carry out orders ,awk -F: ‘{print $1}’ passwd, namely $1, The execution effect is shown in the figure below .

Print the penultimate column , Carry out orders ,awk -F: ‘{print $NF}’ passwd, namely NF, The execution effect is shown in the figure below .

Print the penultimate column , Carry out orders ,awk -F: ‘{print $(NF-1)}’ passwd, namely NF-1, The execution effect is shown in the figure below .

4） Print the specified line
Print page 10-18 That's ok , Carry out orders ,awk -F: ‘{if(NR>=10 && NR<=18){print $1}}’ passwd, The execution effect is shown in the figure below .

5） Add start and end information
Start and end adding descriptive text , Carry out orders ,echo -e “henu\nzknu” | awk ‘BEGIN{print “welcome…”}{print $0}END{print "end… "}’, The execution effect is shown in the figure below .

6） Loop through output printing
loop , Carry out orders ,echo “ Beijing Shanghai Shenzhen Zhumadian " | awk -v str=”" ‘{for(n=1;n<=NF;n++){ str=str dollar n}} END {print str}’, The execution effect is shown in the figure below .

7） Add an empty line and display the line number
Add blank lines , Carry out orders ,sed ‘G’ info.txt, Insert blank lines every other line , The execution effect is shown in the figure below .

Displays the line number of an empty line , Carry out orders ,awk ‘/^dollar/{print}’,NR Indicates line number , The execution effect is shown in the figure below .

Text operation four swordsman （ grep sed awk cut）
grep Used to find matching rows ;
cut Intercept data , Extract calculation by column separator , Not suitable for interception. There are multiple white space characters in the file ;
sed Add or delete check change , Used to intercept by line in text ;
awk Intercept and analyze data , Intercept and analyze data in a vertical column in a file , If there are multiple spaces, it also supports .