Sed、Awk

sed ( SLreamEDitor) Is a powerful and simple text analysis conversion tool , Can read text , And edit the text content according to the specified conditions ( Delete 、 Replace 、 add to 、 Mobile, etc ）, Finally, output all lines or only some lines of processing .sed It can also be done without interaction . To achieve quite complex text processing operations , Widely used in shell Script , It is used to complete various automatic processing tasks .

scd The workflow of mainly includes reading 、 Perform and display three processes .

Read :sed

From the input stream （ file 、 The Conduit 、 The standard input ） Read a line in and store it in a temporary buffer （ Also called pattern space ,patternspace) .

perform : By default , be-all :tcd Commands are executed sequentially in the pattern space , Unless the address of the line is specified , otherwise sced The command will be executed on all lines in turn .

Show : Send the modified content to the output stream . After sending the data , Mode space will be cleared .

Before all the contents of the file are processed , The above process will be repeated , Until all content is processed .

Read a new line into the cache space ;

Extract the first instruction from the specified operation instruction , Judge whether it matches pattern;

If it doesn't match , The subsequent edit commands are ignored , Back to the first 2 Step 2: continue to take out the next instruction ;

If the match , Then the subsequent edit command is executed for the cached row ; After completion , Back to the first 2 Step 2: continue to take out the next instruction ; When all instructions are applied , Output the contents of the cache line ; Back to the first 1 Next, read on to the next line ;

When all the lines have been processed , end ;

Be careful : By default, all sed

Commands are executed in pattern space , So the input file doesn't change , Unless you're using redirection to store output .

1.sed Command common usage

Usually called sed There are two formats for commands , As shown below . among ,“ Parameters ” Refers to the target file of the operation ,

Use when there are multiple operands , Ask the document with a comma “," Separate ; and scriplile Represents the foot wood file , Need to use “-[”

Option assignment , When the script file appears before the target file , Indicates that the input target file is processed through the specified script file .

sed [ Options ] ' operation ' Parameters

sed | Options ] -f  scriptfile Parameters

common sed Command options include the following .

-e or --expression=: Indicates that the input document file is processed with the specified command or if the foot wood .

-f or --file=: Indicates that the input text file is processed with the specified script file ..

-h or --help: Display help .

-n、--quiet or silent: Show only processed results .

-i.bak: Edit the document directly .

-r, -E Using extended regular expressions

-s Treat multiple files as separate files , Instead of a single continuous long file stream

“ operation " Used to specify the action behavior of file operation , That is to say sed The order of . It is usually used “[n1[,n2]]" Format of operation parameters .nl、n2 It's optional , Represents the number of rows selected for operation , If the operation needs to be in 5~ 20 Between lines , Is represented as “5,20

Action action ”. Common operations include the following .

a:  increase , Add a line below the current line to specify .

c:  Replace , Replace the selected row with the specified content .

d:  Delete , Delete selected rows .

i:  Insert , In the selected line . Insert a Line specifies the content .

p:  Print , If you also specify a row , Indicates to print the specified line ; If no line is specified , It means printing everything ; If there are non printing characters , with

ASCII Code output . It is usually associated with “-n” Use options together .

s:  Replace , Replace specified characters .

y:  Character conversion .

// Output everything , Equate to cat test. txt

[ [email protected] localhosl ~]# sed -n ' 3p' test.txt  // Output No 3 That's ok

[ [email protected] ~]# sed -n '3, 5p' test. txt  // Output 3~5 That's ok

[ [email protected] ~]# sed -n 'p;n' test. txt  // Output all odd rows ,n Means to read into the next - Travel information

[ [email protected] ~]# sed -n 'n;p' test. txt  // Output all even lines ,n Means to read into the next - Travel information

[ [email protected] localhosl ~]# 8ed -n '1, 5{p;n}' test.txt   // Output No 1~5 The odd number of lines asked by the line ( The first 1、3、5 That's ok ) he was

short and fat.

[[email protected] ~]# sed -n '10,${n;p]' test.txt   // Output No 10 Even lines from line to file

#woood # AxyzxyzxyzxyzC

Mi sfortunes never come a lone/si ngle.

Case list

ilconfig ena331sed -n 2p # Directly display the second line

In execution “sed -n '10,${n;p}'test.txt" On command , Read the second 1 Line is the... Of the file 10 That's ok , Read the second 2

Line is the... Of the file 11 That's ok , And so on , So the even line of the output is the... Of the file 11 That's ok 、13 Line until the end of the file , It includes the empty line .

Above is sed Basic usage of commands ,sed

When a command is combined with a regular expression , Slightly different format , Regular expression to “/” Surround . for example , The following operations are sed

Examples of commands used with regular expressions .

[[email protected] ~]# sed -n '/the/p' test. txt

// The output contains the The line of

the tongue is boneless but it breaks bones.12! google is the best tools for search keyword.

The year ahead will test our political establi shment to the 1 imit.

[[email protected] ~]# sed -n '4,/.he/p' Lest.txt. // Output from 4 Go first A contain the The line of

the tongue is boneless but it breaks bones.12! google is the best tools for search keyword.

[[email protected] ~]# sed -n ' /the/=' test. txt .

// The output contains the The line number of the line where , Equal sign (=) Used to output line numbers

4

5

(2) Delete eligible text (d)

Because the following examples also need to use test files

test. txt, Therefore, you need to back up the test file before deleting . The following examples demonstrate sed

Several common deletion usages of the command .

In the following order nl The command is used to count the number of lines in a file , Combined with this command, you can see the result of command execution more intuitively .

[ [email protected] ~]# nl test.txt 1 sed '3d' // Delete the first 3 That's ok

1he was short and fat.

2He was wearing a blue polo shirt with black pants.

4Lhe tongue is boneless but it. breaks bones.12!

5google is the best tools for search keyword.

6The year ahead will test our political establishment to the limit.

// Omit part

[ [email protected] localhost ~] 30 nl tost..txt | sod '3, 5d'

// Delete the first 3~5 That's ok

lhe was short and fat.

2lle was wearing a blue po1o shirt with black pants.

6 The year ahead will test our political establ i shment to the limit.

7PT-3.141592653589793238462643383249901429

8a wood cross!

// Omit part

[ [email protected] ~]# nl test. txt |sed ' /cross/d'

// Delete include cross The line of , The first of the logs 8 The line is deleted ; If you want to delete, do not include cross The line of , use ! The symbol indicates the reverse operation ,

Such as ' /cross/ ! d'

....// Omit part

woood t twooo0o0od # AxyzxyzxyzxyzC

I bet this place is really spooky late at night! Misfortunes never come alone/ single.

I shouldn't have lett so tast.

I [email protected] ~] sed '/\.$/d' test.txt / / Delete with ". " The line at the end

the Longue is boneless bul it breaks bones. 12 !

PI=3.141592653589793238462643383249901429

a wood cross !

Actions speak louder than words

#woood

 # #woooooood

# AxyzxyzxyzxyzC

l bet. this place is really spooky late at night!

Be careful : If you delete duplicate blank lines , That is to say, there is only one continuous blank line ,"cat -s test.txt".

(3) Replace the qualified wenmu

In the use of sed Command to replace the operation needs to use s( String substitution )、c( The whole line / Replace the whole piece )、y

( Character conversion ) Command options , The common usage is as follows .

sed 's/the/THE/' Leat.LxL // Put the first one in each line the Replace with THE

sed 's/l/L/2'  test.txt // Place the 2 individual 1 Replace with L

sed ' s/the/ TIlE/g' test. txt // Put all... In the file the Replace with TIE

sed 's/o//g' test.Lxt // Put all... In the file o Delete ( Replace with empty string )

sed 's/^/#/' test.txt // Insert at the beginning of each line # Number

sed ' /the/s/^/#/' test.txt // Include in the Insert at the beginning of each line of # Number

sed 's/$/EOF/' test.txt // Insert a string at the end of each line EOF .

sed '3, 5s/ the/THE/g' test.txt // Will be the first 3~5 All in row the Replace with THE

sed ' /the/s/o/0/g' test.txt // Will include the Of all rows . Replace with 0

sed -i .bak ' s/ SELTNUX-=disabled/ SELTNUX=enable/ ' /otc/sol inux/conf ig

(4) Migrate eligible text .

In the use of sed When the command migrates qualified text , The following parameters are commonly used .

H: Copy to clipboard ;

g、G: Overlay the data in the clipboard / Append to specified line ;

w: Save as a file :

r: Read the specified file ;

a: Add the specified content . The specific operation method is as follows .

I, i Ignore case

sed ' /the/ {H;d};9G' test. txt// take contain the Move to the end of the file ,{;} For multiple operations

sed '1, 5{H;d};17G' test.txt // Will be the first 1~5 Line content transferred to 17 After line

sed ' /the/w out. file' test. txt// Will include the Save as file out. file

sed '/the/r /etc/hostname' test. txt // Will file /etc/hostname Add the content of to include the After each line of

sed '3aNew' test.txt // In the 3 Insert a... After the line New line , The content is New

sed ' /the/aNcw' test. txt.// Include in the Insert a new row after each row of , The content is New

sed '3aNew1\nNew2' test.txt // In the 3 Insert multiline content after row , In the middle of the \n Means line break

(5) Editing files with scripts

Use sed The script stores multiple editing instructions in a file ( One editing instruction per line ), adopt “-I” Option to call . For example, execute the following command to convert the second 1~5 Line content transferred to 16 After line .

sed '1,5{H;d};16G' test.txt // Will be the first 1~5 Line content transferred to 16 After line

The above operations can be done in script file mode :

[ [email protected] ~]# vi opt.list

1, 5H

1, 5d

16G

[ [email protected] ~]# sed -f opt.list test. txt

The year ahead will test our political establishment to the limit.

PI=3.141592 653589793238462643383249901429

a wood cross!

AcL ions speak louder Lhan words

#woood # #woooooood # AxyzxyZXyZXyZC

bet. this place is real ly spooky late at night! Misfortunes never come a lonc/singlc.

I shouldn't have lett so tast.

he was short. and fat.

He was wearing a blue polo shirt with black pants. The home of Football on BBC Sport online.

the tongue is boneless but it breaks bones.12! google is the best tools for search keyword.

|| Indicates the last - After a command fails to execute , To execute the next command

(6) sed Direct manipulation File example

Write a script , Used to adjust vsftpd Service configuration , Require anonymous users to be banned , But allow local users to ( It is also allowed to write ).

[[email protected] calhost ~]f vim local_ only_ ftp. sh

#! /bin/bash

# Specify sample file path 、 Profile path

SAMPLE ="/usr/ share/doc/vsftpd-3.0.2/EXAMPLE/ INTERNET_ S1TE/ vsftpd.conf "

CONFIG= " /etc/vsftpd/vsftpd. conf"

# Back up the original profile , The test file name is /etc/vsttpd/vsttpd.cont.bak Whether the backup file exists , If not, use cp Command for file backup

[ ! -e"SCONF1G.bak" ] && cp SCONrIG SCoNr1G.bak #f Adjust based on sample configuration , Overlay existing files

sed -e ' /^anonymous_ enable/s/ YES/NO/g' SSAMPLE > SCONFIG

sed -i -e ' /^1ocal_ enab1e/s/NO/ YES/g' -e ' /^write_ enab1e/s/NO/ YES/g' SCONFIG

grep "listen" SCONFIG || sed -i 'Salisten=-YES' SCONFIG

And start the vsftpd service , And it is set to run automatically after power on

systemctl restart vsftpd systemctl enable vsftpd

[ root @localhost ~]# chmod +x local_ only_ ftp.sh

awk

summary :

AWK Is a language for processing text files , Is a powerful text analysis tool .

It is a programming language specially designed for text processing , It's also line processing software , Usually used to scan 、 Filter 、 Statistical summary work data can be from standard input, pipeline or this

20 century 70 Was born in Bell Laboratories , Now? centos7 It's using gawk

The name AWK、 Because it took three founders Alfred Aho,Peter Meinberger, and Brian Kernighan Of Family Mame First character .

working principle :

When you read the first line , Matching condition , Then perform the specified action , Then read the second line of data processing , No default output

If no matching criteria are defined, the default is to match all data rows ,awk Implicit loop , The action will be executed as many times as the condition matches

working principle :

Read the text line by line , The default is space or tab Key to separate the separator , Save the separated fields to the built-in variables , And execute the edit command according to the mode or condition .

sed Commands are often used for a whole line of processing , and awk They tend to divide a line into multiple "" Field " And then deal with it .awk Information is also read line by line , The execution result can be obtained through print To print and display field data . In the use of awk In the course of the order , You can use logical operators "

&" Express " And "、"I Express " or "、"!" Express non "; You can also do simple mathematical operations , Such as +、 One 、+、/、%、^ Respectively means plus 、 reduce 、 ride 、 except 、 The remainder and the power .

Command format :

awk Options · Mode or condition { operation }" file 1 file 2 ...

awk -f Script files    file 1     file 2 ..

Format : awk keyword     Options     Command part   '{xxxx} '    file name

awk Contains several special built-in variables （( Can be directly used ） As shown below :

FS: Specify the field separator for each line of text , Default to space or tab stop .

NF: Number of fields in the row currently processed .

NR: Line number of the currently processed line ( Yu Shu ).

$o: The entire contents of the currently processed row .

$n: Of the current processing line n A field ( The first n Column ).

FLENAME: File name processed .

Rs: Line separator .awk When reading from a file , Based on the R.s The definition of cut data into many records , and awk Read only one record at a time , To deal with . The default is ” \n'

Jane said : Data record separation , The default is \n, One record for each action

awk Common built-in variables : $1、 $2、NF、NR、$0

$1: On behalf of the - - Column

$2: Represents the second column, and so on

$0: For the whole line

NF: The number of columns in a row ( It also contains the last column )

NR: Row number

Line by line what tasks to perform before starting , What tasks will be performed after the end of , use BEGIN、END

BEGIN- Generally used for initialization , Read the data only once before

END It is generally used for summary operation , Execute only once after reading the data record - Time

Usage of other built-in variables FS、OFs、NR、FNR、RS、ORS

[ [email protected] ~]# awk ' BEGIN{FS=":"}{print $1}' pass. Txt // Define the field separator as colon... Before printing

root

bin

da emon

adm

1p

awk Advanced usage

Define reference variables

[[email protected] ~]# a=100

[[email protected] ~]# awk -v b="$a" 'BEGIN{print b} '  // Set the system variables a, stay awk Assign a value to a variable b, And then call variables. b

100

[[email protected] ~]# awk 'BEGIN{print "'$a'"}' // If you call directly, you need to use double quotation marks first and then single quotation marks Number

100

[[email protected] ~]# awk -vc=1 'BEGIN{print c}' //awk Direct determination Define variables and reference

Call function getline, When reading a row of data, you do not get the current row, but the next row of the current row -- That's ok

[ [email protected] localhost ~]# df -h | awk ' BEGIN{getline}/root/ {print $0} '

/ dev/mapper/centos-root

50G

5.2G

45G

11 Number /

[[email protected] ~]# seq 10 | awk ' {getline;print $0}' // Show even rows

2

4

6

8

10

[[email protected] ~]# seq 10 |awk ' {print $O;getline}' // Show odd rows

1

3

5

7

9

if sentence : awk Of if Statements are also divided into single branches 、 Two branches and many branches

Single branch is if( ) { }

The two branches are if( ) { }else{ }

Multiple branches are if( ){ }else if( ) { }else{ }

[ [email protected] ~]# awk -F: ' {if($3<10) {print $0}}' /etc/pas swd

// The third column is less than 10 Print entire line

[[email protected] ~]# awk -F: '{if($3<10) {print $3}else{print $1}}' /etc/ pas swd

// The third column is less than 10 Print the third column , Otherwise, print the first column

awk And support for loop 、while loop 、 function 、 Array etc.

Through pipes 、 Double quotes call shell command :

echo $PATH | awk 'BEGIN{RS=":"};END {print NR} '  # Count the number of colon separated text paragraphs ,END{

} In the block , Often put in the print results and other statements

echo $PATH | awk ' BEGIN{RS=":"}; {print NR,$0} ;END {print NR} '