Shell (13) Three Musketeers

Shell（13） Three swordsmen

Preface

Shell The three main commands in grep、sed、awk, because Linux Everything is the nature of the document , So these three commands have powerful functions in text editing , And widely used , Known by insiders as "shell Three swordsmen ".

One 、grep command

The basic format ：grep [ Options ]… Search for conditions Target file

Options	meaning
-E	Open extension （Extend） Regular expression of
-c	Calculate find ‘ Search string ’ The number of times
-i	Ignore case differences , So case is the same
-o	Only the string matched by the pattern
-v	Reverse selection , That is to say, no ‘ Search string ’ Content line ！（ Reverse lookup , Output lines that do not match the search criteria ）
–color=auto	You can add color to the key words you find
-n	Output line number

grep The extended command for ——egrep command

grep Command only supports basic regular expressions , If you use extended regular expressions , Need to use egrep .egrep Command and grep The usage of commands is basically similar .egrep Command is a search file acquisition mode , Use this command to search for any string and symbol in the file , You can also search for a string of one or more files , A prompt can be a single character 、 A string 、 A word or sentence .

 Example ：
egrep -n 'wo+d' test.txt  # Inquire about "wood" "woood" "woooooood" Etc 
egrep -n 'of|if|on' test.txt  # Inquire about "of" perhaps "if" perhaps "on" character string 
egrep -n 't(a|e)st' test.txt  #“tast” And “test” Because of these two words “t” And “st” It's repetitive , So will “a” And “e” Listed in “()” Among the symbols , And “|” Separate , You can query "tast" perhaps "test" character string 
egrep -n 'A(xyz)+C' test.txt  # The command is at the beginning of the query "A" It ends with "C", More than one in the middle "xyz" Meaning of string 

Two 、sed command

sed（Stream EDitor） Is a powerful and simple text analysis conversion tool , Can read text , And edit the text content according to the specified conditions （ Delete 、 Replace 、 add to 、 Mobile, etc ）, Finally, output all lines or only some lines of processing .sed It can also realize quite complex text processing operation without interaction , Widely used in Shell Script , It is used to complete various automatic processing tasks . By default, all sed Commands are executed in pattern space , So the input file doesn't change , Unless you're using redirection to store output .

sed The workflow of mainly includes reading 、 Perform and display three processes .

· Read ：sed From the input stream （ file 、 The Conduit 、 The standard input ） Read a line in and store it in a temporary buffer （ Also called pattern space ,pattern space）.

· perform ： By default , be-all sed Commands are executed sequentially in pattern space , Unless the address of the line is specified , otherwise sed The command will be executed on all lines in turn .

· Show ： Send the modified content to the output stream . After sending the data , Mode space will be cleared .
Before all the contents of the file are processed , The above process will be repeated , Until all content is processed .

sed Command common usage

Usually called sed There are two formats for commands , As shown below . among ,“ Parameters ” Refers to the target file of the operation , Use when there are multiple operands , Comma between files “,” Separate ; and scriptfile Represents a script file , Need to use “-f” Option assignment , When the script file appears before the target file , Indicates that the input target file is processed through the specified script file .

The basic format ：

sed [ Options ] ' operation '  Parameters 
sed [ Options ] -f scriptfile  Parameters 

Options	meaning
-e or –expression	Represents a text file that uses a specified command or script to process input
-f or –file	Indicates that the input text file is processed with the specified script file
-h or –help	Display help
-n、–quiet or silent	Show only processed results
-i	Edit text file directly
-r or -E	Using extended regular expressions
-s	Treat multiple files as separate files , Instead of a single continuous long file stream

“ operation ” Used to specify the action behavior of file operation , That is to say sed The order of . It is usually used “[n1,n2]” Format of operation parameters .n1、n2 It's optional , Represents the number of rows selected for operation , If the operation needs to be in 5～ 20 Between lines , Is represented as “5,20 Action action ”. Common operations include the following .

operation	meaning
a	increase , Add a line below the current line to specify
c	Replace , Replace the selected row with the specified content
d	Delete , Delete selected rows
i	Insert , Insert a row above the selected row to specify
p	Print , If you also specify a row , Indicates to print the specified line ; If no line is specified , It means printing everything ; If there are non printing characters , with ASCII Code output . It is usually associated with “-n” Use options together
s	Replace , Replace specified characters
y	Character conversion

 Example ： Output eligible text （p  Indicates normal output ）
 sed -n 'p' test.txt    # Output everything , Equate to  cat test.txt
sed -n '3p' test.txt	# Output No  3  That's ok 
sed -n '3,5p' test.txt	# Output  3~5  That's ok 
sed -n 'p;n' test.txt	# Output all odd rows ,n  Read in the next line 
sed -n 'n;p' test.txt	# Output all even lines ,n  Read in the next line 
sed -n '1,5{p;n}' test.txt   # Output No  1~5  Odd rows between rows ( The first  1、3、5  That's ok ) 
sed -n '10,${n;p}' test.txt  # Output No  10  Even lines from line to end of file 

In execution “sed -n‘10,${n;p}’test.txt” On command , Read the second 1 Line is the... Of the file 10 That's ok , Read the second 2 Line is the... Of the file 11 That's ok , And so on , So the even line of the output is the... Of the file 11 That's ok 、13 Line until the end of the file , It includes the empty line .

Above is sed Basic usage of commands ,sed When a command is combined with a regular expression , Slightly different format , Regular expression to “/” Surround . for example , The following operations are sed Examples of commands used with regular expressions .

sed -n '/the/p' test.txt	# The output contains the  The line of 
sed -n '4,/the/p' test.txt	# Output from  4  Line to the first containing  the  The line of 
sed -n '/the/=' test.txt  # The output contains the  The line number of the line where , Equal sign (=) Used to output line numbers 
sed -n '/^PI/p' test.txt  # Output to PI  Beginning line 
sed -n '/[0-9]$/p' test.txt	 # Output lines ending in numbers 
sed -n '/\<wood\>/p' test.txt   # Output contains words wood  The line of ,\<、\> For word boundaries 

 Example ： Delete eligible text （d）
sed '3d'	# Delete the first  3  That's ok 
sed '3,5d'	# Delete the first  3~5  That's ok 
sed '/cross/d'  # Delete include  cross  The line of , Original paragraph  8  The line is deleted ; If you want to delete, do not include  cross  The line of , use ! The symbol indicates the reverse operation ,  Such as '/cross/！d'
sed '/^[a-z]/d' test.txt	# Delete lines that start with lowercase letters 
 sed '/\.$/d' test.txt	# Delete with "." The line at the end 
 sed '/^$/d' test.txt	# Delete all blank lines , If you delete duplicate blank lines , That is to say, there is only one continuous blank line 

In the use of sed Command to replace the operation needs to use s（ String substitution ）、c（ The whole line / Replace the whole piece ）、y（ Character conversion ） Command options , The common usage is as follows .

 Example ： Replace eligible text 
sed 's/the/THE/' test.txt	# First in each line the  Replace with  THE
sed 's/l/L/2' test.txt   	# Place the  2  individual  l  Replace with  L
sed 's/the/THE/g' test.txt	# Put all... In the file the  Replace with  THE
sed 's/o//g' test.txt	    # Put all... In the file o  Delete ( Replace with empty string )
sed 's/^/#/' test.txt	    # Insert at the beginning of each line # Number 
sed '/the/s/^/#/' test.txt	# Include in the  Insert at the beginning of each line of # Number 
sed 's/$/EOF/' test.txt     # Insert a string at the end of each line EOF
sed '3,5s/the/THE/g' test.txt	# Will be the first  3~5  All in row  the  Replace with  THE
sed '/the/s/o/O/g' test.txt	# Will include the  Of all rows  o  Replace with  O

Migrate parameters of qualified text ：

Parameters	meaning
H	Copy to clipboard
g、G	Overlay the data in the clipboard / Append to specified line
w	Save as a file
r	Read the specified file
a	Add the specified content . The specific operation method is as follows
I、i	Ignore case

sed '/the/{H;d};$G' test.txt	# Will include the  Move to the end of the file ,{;} For multiple operations 
sed '1,5{H;d};17G' test.txt	    # Will be the first  1~5  Line content transferred to  17  After line 
sed '/the/w out.file' test.txt	# Will include the  Save as file  out.file
sed '/the/r /etc/hostname' test.txt	# Will file /etc/hostname  Add the content of to include  the  After each line of 
sed '3aNew' test.txt	        # In the  3  Insert a new row after row , The content is New
sed '/the/aNew' test.txt	    # Include in the  Insert a new row after each row of , The content is  New
sed '3aNew1\nNew2' test.txt	    # In the  3  Insert multiline content after row , In the middle of the \n  Means line break 

3、 ... and 、awk command

AWK Is a language for processing text files , Is a powerful text analysis tool . It is a programming language specially designed for text processing , It's also line processing software , Usually used to scan 、 Filter 、 Statistical summary work , The data can come from standard input, pipe or file .

20 century 70 Was born in Bell Laboratories , Now? centos7 It's using gawk The name AWK Because it took three founders Alfred Aho,Peter Weinberger, and Brian Kernighan Of Family Name First character .

working principle :

Read the text line by line , The default is space or tab Key to separate the separator , Save the separated fields to the built-in variables , And execute the edit command according to the mode or condition .

sed Commands are often used for a whole line of processing , and awk They tend to divide a line into multiple "“ Field " And then deal with it .awk Information is also read line by line , The execution result can be obtained through print To print and display field data . In the use of awk In the course of the order , You can use logical operators ” &&“ Express " And ”、“|| Express " or ”、"!" Express non , You can also do simple mathematical operations , Such as +、 One 、*、/、%、^ Respectively means plus 、 reduce 、 ride 、 except 、 The remainder and the power .

Command format :

awk Options ' Mode or condition { operation }' file 1 file 2 ...
awk -f  Script files   file 1   file 2 ..
awk keyword      Options        Command part   '{xxxx}'    file name 

awk Contains several special built-in variables （ Can be directly used ） As shown below ：

Built in variables	meaning
FS	Specify the field separator for each line of text , Default to space or tab stop
NF	Number of fields in the row currently processed
NR	Line number of the currently processed line （ Ordinal number ）
$0	The entire contents of the currently processed row
$n	Of the current processing line n A field （ The first n Column ）
FILENAME	File name processed
RS	Line separator , Data record separation , The default is \n, One record for each action

awk Execute line by line in the command. What tasks should be executed before starting , What tasks will be performed after the end of , Then use BEGIN、END

BEGIN： Generally used for initialization , Read the data only once before

END： It is generally used for summary operation , Execute only once after reading the data record

awk Arithmetic ：

awk 'BEGIN{x=10;print x}'  # If you don't use quotation marks awk Just output it as a variable , So there's no need to add $ 了 
10
awk 'BEGIN{x=10;print x+1}'  #BEGIN Before processing files , Therefore, not following the file name does not affect 
11
awk 'BEGIN{print x+1}'    # Do not specify the initial value , The initial value is 0, If it's a string , The default value is null 
1
awk -F: '$1~/root/' /etc/passwd  # Fuzzy matching , use ~ Means to contain ,!~ Does not include 

About the comparison between numeric value and string ：

Comparative symbols ：== != <= >= < >

awk  'NR==5{print}' /etc/passwd  # Print the fifth line 

Usage of other built-in variables FS、OFS、NR、FNR、RS、ORS

awk 'BEGIN{FS=":"}{print $1}' pass.txt  # Define the field separator as colon... Before printing 
awk 'BEGIN{FS=":";OFS="---"}{print $1,$2}' pass.txt   #OFS Defines how the output is separated ,$1$2 The middle should be separated by commas , Because commas are mapped to by default OFS Variable , This variable defaults to a space 
awk '{print FNR,$0}' /etc/resolv.conf /etc/hosts   # It can be seen that FNR The line number of is appended when there are multiple files 
awk 'BEGIN{RS=":"}{print $0}' /etc/passwd   #RS: Specify what to use as the newline character , The colon specified here , What you specified must be the characters in the original text 
awk 'BEGIN{ORS=" "}{print $0}' /etc/passwd   # Combine multiple lines into one line and output , When outputting, customize to separate each line with a space , The default is enter 

END