8. SHELL file processing Three Musketeers sed

One ,sed The role of

sed（Stream EDitor） Is a powerful and simple text analysis conversion tool , Can read text , And edit the text content according to the specified conditions （ Delete 、 Replace 、 add to 、 Mobile, etc ）, Finally, output all lines or only some lines of processing .sed It can also realize quite complex text processing operation without interaction , Widely used in Shell Script , It is used to complete various automatic processing tasks

1.1 Workflow

• sed The workflow of mainly includes reading 、 Perform and display three processes

Read ：
sed From the input stream （ file 、 The Conduit 、 The standard input ） Read a line in and store it in a temporary buffer （ Also called pattern space ,pattern space）

perform ：
By default , be-all sed Commands are executed sequentially in pattern space , Unless the address of the line is specified , otherwise sed The command will be executed on all lines in turn

Show ：
Send the modified content to the output stream . After sending the data , Mode space will be cleared
Before all the contents of the file are processed , The above process will be repeated , Until all content is processed

Be careful ： By default, all sed Commands are executed in pattern space , So the input file doesn't change , Unless you're using redirection to store output .

Two ,sed Command common usage

• Usually called sed There are two formats for commands

1.sed [ Options ] ‘ operation ’ Parameters

2.sed [ Options ] -f scriptfile Parameters
“ Parameters ” Refers to the target file of the operation , Use when there are multiple operands , Comma between files “,” Separate ;
and scriptfile Represents a script file , Need to use “-f” Option assignment , When the script file appears before the target file , Indicates that the input target file is processed through the specified script file

2.1 common sed Command options

Options	meaning
-f or –file=	Indicates that the input text file is processed with the specified script file
-n –quiet or silent	Show only processed results
-i	Edit text file directly
-r, -E	Using extended regular expressions
-s	Treat multiple files as separate files , Instead of a single continuous long file stream
-h or –help	Display help
-e or –expression=	Represents a text file that uses a specified command or script to process input

2.2 Common operations

operation ” Used to specify the action behavior of file operation , That is to say sed The order of . It is usually used “[n1[,n2]]” Format of operation parameters .n1、n2 It's optional , Represents the number of rows selected for operation , If the operation needs to be in 5～ 20 Between lines , Is represented as “5,20 Action action

Parameters	meaning
a	increase , Add a line below the current line to specify
c	Replace , Replace the selected row with the specified content
d	Delete , Delete selected rows
i	Insert , Insert a row above the selected row to specify
p	Print , If you also specify a row , Indicates to print the specified line ; If no line is specified , It means printing everything ; If there are non printing characters , with ASCII Code output . It is usually associated with “-n” Use options together
s	Replace , Replace specified characters
y	Character conversion

3. Basic usage example

3.1 Output eligible text （p Indicates normal output ）

file i Content

[[email protected] ky20]# cat i ## The line number is in front 
  1 320102199106101111
  2 tet
  3 test
  4 teat
  5 atelt
  6 tesalt
  7 tselt
  8 tsalt
  9 teet
 10 tseset
 11 tesest
 12 tealt
 13 tsst
 14 teesst
 15 teslt
 16 1 2 3 4 5
 17 1 2 3 4 5
 18 2
 19 3
 20 4
 21 5

1. Look at the output line

[[email protected] ky20]# cat test | sed -n '3p'
test

2. Check the number of output lines

[[email protected] ky20]# cat test | sed -n '3,5p'
 test
 teat
 atelt

3. Output all odd rows ,n Read in the next line ,p Means to print

[[email protected] ky20]# sed -n 'p;n' i
1
3
5

4. Output all even lines ,n Read in the next line

[[email protected] ky20]# sed -n 'n;p' i
2
4

5. Output No 1~5 Odd rows between rows ( The first 1、3、5 That's ok ) The content of

[[email protected] ky20]# sed -n '1,5{n;p}' i
tet
teat
tesalt

6. Output No 10 Even lines from line to end of file

In execution “sed -n‘10,${n;p}’ i” On command , Read the second 1 Line is the... Of the file 10 That's ok , Read the second 2
Line is the... Of the file 11 That's ok , And so on , So the even line of the output is the... Of the file 11 That's ok 、13 Line until the end of the file , It includes the empty line

[[email protected] ky20]# sed -n '10,${n;p}' i
tesest
tsst
teslt
1 2 3 4 5
3
5

7. Directly display the second line

[[email protected] ky20]# ifconfig ens33|sed -n 2p 
        inet 192.168.113.126  netmask 255.255.255.0  broadcast 192.168.113.255

8. The output contains ens33 The line of ,test Is the network card information

[[email protected] ky20]# sed -n '/ens33/p' test
NAME=ens33
DEVICE=ens33

9. Output from 10 The first one at the beginning of the line contains ens33 The line of

[[email protected] ky20]# sed -n '10,/ens33/p' test
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=ens33

10. The output contains ens33 The line number of the line where , Equal sign (=) Used to output line numbers

[[email protected] ky20]# sed -n '/ens33/=' test
12
14

11. Output to I Beginning line

[[email protected] ky20]# sed -n '/^I/p' test
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes

12. Output lines ending in numbers

[[email protected] ky20]# sed -n '/[0-9]$/p' test
NAME=ens33
UUID=548f4e9b-9958-4ae2-b61c-0dbc15f3e7a8
DEVICE=ens33

13. Output contains words ens33 The line of ,<、> For word boundaries ,,<、> Have special meaning to transfer

[[email protected] ky20]# sed -n '/\<ens33\>/p' test
NAME=ens33
DEVICE=ens33

3.2 Delete eligible text （d Delete , Delete selected rows ）

• sed Several common deletion usages of the command
• In the following order nl The command is used to count the number of lines in a file , Combined with this command, you can see the result of command execution more intuitively

1. Delete the first 3 That's ok

[[email protected] ky20]# cat i
320102199106101111
tet
test
teat
[[email protected] ky20]# nl i |sed '3d'
     1	320102199106101111
     2	tet
     4	teat

2. Delete the first 3 To 5 That's ok

[[email protected] ky20]# nl i |sed '3,5d'
     1	320102199106101111
     2	tet
     6	tesalt

3. Delete include tes The line of , If you want to delete, do not include cross The line of , use ! The symbol indicates the reverse operation , Such as ’/cross/！d’

[[email protected] ky20]# nl i |sed '/tes/d'
     1	320102199106101111
     2	tet
     4	teat
     5	atelt
     7	tselt

4. Delete lines that start with lowercase letters

[[email protected] ky20]# sed '/^[a-z]/d' i
320102199106101111
1 2 3 4 5
1 2 3 4 5

5. Delete with "." The line at the end

[[email protected] ky20]# sed '/\.$/d' test.txt

6. Delete all blank lines

[[email protected] ky20]# sed '/^$/d' test.txt

notes It means ： if yes Delete except heavy complex Of Blank line , namely even To continue Of empty That's ok only Protect leave One individual , “cat -s i” ,cat -s Do not output multiple blank lines

3.3 Replace eligible text ***

In the use of sed Command to replace the operation needs to use

s（ String substitution ）

c（ The whole line / Replace the whole piece ）

y（ Character conversion ） Command options

Common usage ：

1. First in each line tet Replace with aaa

[[email protected] ky20]# sed 's/tet/aaa/' i
320102199106101111
aaa
test aaa
aaaat tet
atelt aaa

2. Place the 2 individual tet Replace with aaa

[[email protected] ky20]# sed 's/tet/aaa/2' i
320102199106101111
tet
test tet
tetat aaa

3. Put all... In the file tet Replace with AAA

[[email protected] ky20]# sed 's/tet/AAA/g' i
320102199106101111
AAA
test AAA
AAAat AAA

4. Put all... In the file o Delete ( Replace with empty string )

[[email protected] ky20]# sed 's/t//g' i
320102199106101111
e
es e
ea e

5. Insert at the beginning of each line # Number

[[email protected] ky20]# sed 's/^/#/g' i
#320102199106101111
#tet

6. contain tet Insert at the beginning of each line of # Number

[[email protected] ky20]# sed '/tet/s/^/#/g' i
320102199106101111
#tet
#test tet

7. Insert a string at the end of each line EOF

[[email protected] ky20]# sed 's/$/ AAA/g' i
320102199106101111 AAA
tet AAA
test tet AAA

8. Will be the first 3~5 All in row t Replace with T

[[email protected] ky20]# sed '3,5s/t/T/g' i
320102199106101111
tet
TesT TeT
TeTaT TeT
aTelT TeT

9. Will include tet Of all rows a Replace with Q

[[email protected] ky20]# sed '/tet/s/a/Q/g' i
320102199106101111
tet
test tet
tetQt tet
Qtelt tet

3.4 Migrate eligible text

In the use of sed When the command migrates qualified text , The following parameters are commonly used
H: Copy to clipboard
g、G: Overlay the data in the clipboard / Append to specified line
w： Save as a file
r： Read the specified file
a： Add the specified content . The specific operation method is as follows
I,i Ignore case

1. Will include tet Move to the end of the file ,{;} For multiple operations , Try it yourself for too long

[[email protected] ky20]# sed '/tet/{H;d};$G' i

2. Will be the first 1~5 Line content transferred to 17 After line , Try it yourself for too long

[[email protected] ky20]# sed '1,5{H;d};17G' i

3. Will include tet Save as file q.txt

[[email protected] ky20]# sed '/tet/w q.txt' i

[[email protected] ky20]# ls
access_log  i  passwd  q.txt  shadow  test

[[email protected] ky20]# cat q.txt 
tet
test tet
tetat tet

4. Will file /etc/hostname Add the content of to include tet After each line of

[[email protected] ky20]# sed '/tet/r /etc/hostname' i
320102199106101111
tet
qjm
test tet
qjm

5. In the 3 Insert a new row after row , The content is New

[[email protected] ky20]# sed '3aNew' i
320102199106101111
tet
test tet
New

6. Include in tet Insert a new row after each row of , The content is New

[[email protected] ky20]# sed '/tet/aNew' i
[[email protected] ky20]# sed '/tet/aNew' i
320102199106101111
tet
New
test tet
New

7. In the 3 Insert multiline content after row , In the middle of the \n Means line break

[[email protected] ky20]# sed '3aNew1\nNew2' i
320102199106101111
tet
test tet
New1
New2

notes ： If you want the above operations to take effect in the original document , Add - i that will do

5. In the 3 Insert a new row after row , The content is New

[[email protected] ky20]# sed '3aNew' i
320102199106101111
tet
test tet
New

6. Include in tet Insert a new row after each row of , The content is New

[[email protected] ky20]# sed '/tet/aNew' i
[[email protected] ky20]# sed '/tet/aNew' i
320102199106101111
tet
New
test tet
New

7. In the 3 Insert multiline content after row , In the middle of the \n Means line break

[[email protected] ky20]# sed '3aNew1\nNew2' i
320102199106101111
tet
test tet
New1
New2

notes ： If you want the above operations to take effect in the original document , Add - i that will do