Awk of shell script

Preface ： The anchor is in a bad mood these days , So there is no preface , But as the eldest brother of the Three Musketeers ,awk Is the most difficult and important , Friends of abduction should take it seriously

Catalog

One .AWK Introduction to

1.1AWK Overview

1.2AWK How it works

1.3AWK Command basic format

1.4AWK Built in variables for

2.AWK Examples

2.1 Display the number of lines and print

2.2 Two ways to view

2.3 Separated by colon , Print the first column

 2.4 With x As a separator ​ edit

2.5 Print the contents of the first and second columns at the same time

 2.6 Print the contents of the first and last columns

2.7 Split by tab

2.8 With ：/ For division , Print the ninth column

 2.9 Print disk usage

 2.10 Print string

2.11  Print a string to determine how many lines there are in the file

3.awk Common built-in variables

3.1 experiment

3.1.1 The print contains root The whole line of

 3.1.2 The print contains root The first column of the row

3.1.3 The print contains root The first and sixth columns of the row

3.1.4 Show root The total number of columns in that row

 3.1.5  With ：/ For division , The print contains root The number of rows

3.1.6 Print the total number of rows

 3.1.7 Print the rows and columns

 3.1.8 Print the last column

 3.1.9 Print the last line of the file

 3.1.10 Print the number of columns per row

 3.1.11  According to the line Numbers

 3.1.12 Print the second line

3.1.13 Print how many columns there are in the current row

3.1.14 Expand production ： NIC ip、 Traffic

 3.1.15 awk Arithmetic

 3.1.16  Comparison of numeric value and string

 3.2.1 Print the fifth line

 3.2.2 Print lines <5 The content of

 3.2.3 The first column of exact matching and fuzzy matching is root Of

 3.2.4 The third column >=1000 Of

 3.2.5 Logical operations && ||

3.3 Print a list of numbers

 4. Other built-in variables

4.1 Define the field separator as colon before printing

 4.2//OFS Defines how the output is separated

 4.3 FNR The line number of is appended when there are multiple files

 4.4RS: Specify what to use as the newline character , The colon specified here , What you specified must be the characters in the original text

4.5 Combine multiple lines into one line and output

 4.6 awk Advanced usage of

 4.6.1 Define reference variables

 4.6.2 Call function getline, When reading a row of data, you do not get the current row, but the next row of the current row

4.6.3 Show odd and even rows

  5、 ... and .awk Of if sentence Single branch Multiple branches Double branch

  6、 ... and . summary

---

One .AWK Introduction to

1.1AWK Overview

AWK Is a language for processing text files , Is a powerful text analysis tool .
It is a programming language specially designed for text processing , It's also line processing software , Usually used to scan 、 Filter 、 Statistical summary work
The data can come from standard input, pipe or file
20 century 70 Was born in Bell Laboratories , Now? centos7 It's using gawk

1.2AWK How it works

1. When you read the first line , Matching condition , Then perform the specified action , Then read the second line of data processing , No default output , If no matching criteria are defined, the default is to match all data rows ,awk Implicit loop , The action will be executed as many times as the condition matches

2. Read the text line by line , The default is space or tab Key to separate the separator , Save the separated fields to the built-in variables , And execute the edit command according to the mode or condition .

3.sed Commands are often used for a whole line of processing , and awk Compare 、 Tends to divide a line into multiple "" Field " And then deal with it .awk Information is also read line by line , The execution result can be obtained through print To print and display field data . In the use of awk In the course of the order , You can use logical operators " &&" Express " And "、"|| Express " or "、"!" Express non "; You can also do simple mathematical operations , Such as +、 One 、*、/、%、^ Respectively means plus 、 reduce 、 ride 、 except 、 The remainder and the power .

1.3AWK Command basic format

1.awk Options ' Mode or condition { operation }' file 1 file 2 ...
2.awk -f Script files file 1   file 2 ..

Format ：awk keyword     Options       Command part  '{xxxx}'   file name

1.4AWK Built in variables for

Built in variable command	Meaning of built-in variables
FS	Specify the fields for each line of text Separator , Default to space or tab stop .
NF	Number of fields in the row currently processed .
NR	Line number of the currently processed line （ Ordinal number ）.
$0	The entire contents of the currently processed row .
$n	Of the current processing line n A field （ The first n Column ）.
FILENAME	File name processed .
RS	Line separator .awk When reading from a file , Based on the RS The definition of cut data into many records , and awk Read only one record at a time , To deal with . The default is ’\n’
FNR	Record number （ Line number ）, from 1 Start , The new file comes back from 1 Start counting

ps：awk When reading from a file , Based on the RS The definition of cut data into many records , and awk Read only one record at a time , To deal with . The default is ’\n’
Jane said ： Data record separation , The default is \n, One record for each action .

2.AWK Examples

2.1 Display the number of lines and print

[[email protected] ~]# cat /etc/passwd | head -10 > hehe
[[email protected] ~]# cat hehe

2.2 Two ways to view

2.3 Separated by colon , Print the first column

 2.4 With x As a separator

2.5 Print the contents of the first and second columns at the same time

 2.6 Print the contents of the first and last columns

2.7 Split by tab

2.8 With ：/ For division , Print the ninth column

 2.9 Print disk usage

 2.10 Print string

The difference between double quotation marks and no double quotation marks

2.11  Print a string to determine how many lines there are in the file

[[email protected] ~]# awk '{print "hello"}' /etc/fstab 
hehe
hehe
hehe
hehe
hehe
hehe
hehe
hehe
hehe
hehe
hehe
hehe
[[email protected] ~]# awk '{print "hello"}' /etc/fstab |wc -l
12
[[email protected] ~]# cat /etc/fstab |wc -l
12

3.awk Common built-in variables

command	meaning
$1	For the first column
$2	Represents the second column, and so on
$0	For the whole line
NF	The number of columns in a row
NR	Row number

3.1 experiment

3.1.1 The print contains root The whole line of

[[email protected] awk]# awk -F: ‘/root/{print $0}’ zz
root：x:0:0:root:/root:/bin/bash
operator：x:11:0:operator:/root:/sbin/nologin

 

 3.1.2 The print contains root The first column of the row

[[email protected] awk]# awk -F: ‘/root/{print $1}’ zz
root
operator

3.1.3 The print contains root The first and sixth columns of the row

[[email protected] ~]# awk -F: ‘/root/{print $1,$6}’ pass.txt // The print contains root The first and sixth columns of the row of

 

3.1.4 Show root The total number of columns in that row

 3.1.5  With ：/ For division , The print contains root The number of rows

3.1.6 Print the total number of rows

 3.1.7 Print the rows and columns

 [[email protected] ~]# awk -F: ‘NR==2{print $1}’ /etc/passwd // Print the first column of the second row

 3.1.8 Print the last column

 3.1.9 Print the last line of the file

[[email protected] ~]# awk ‘END{print $0}’ /etc/passwd // Print the last line of the file

 3.1.10 Print the number of columns per row

 3.1.11  According to the line Numbers

 3.1.12 Print the second line

[[email protected] ~]# awk ‘NR==2’ /etc/passwd
// Print the second line , No addition print It's the same thing , The default is to print bin：x:1:1:bin:/bin:/sbin/nologin

[[email protected] ~]# awk ‘NR==2{print}’ /etc/passwd // Same as above
bin：x:1:1:bin:/bin:/sbin/nologin

3.1.13 Print how many columns there are in the current row

[[email protected] ~]# awk  -F: '{print " The current line has "NF" Column "}' zz
 The current line has 7 Column 
 The current line has 7 Column 

3.1.14 Expand production ： NIC ip、 Traffic

1. Line by line what tasks to perform before starting , What tasks will be performed after the end of , use BEGIN,END

2.BEGIN Generally used for initialization , Read the data only once before

3.END It is generally used for summary operation , Execute only once after reading the data record

[[email protected] ~]# ifconfig ens33 | awk '/netmask/{print " Native ip The address is "$2}'
 Native ip The address is 192.168.75.135

[[email protected] ~]# ifconfig ens33 | awk '/RX p/{print $5" byte "}'
8341053 byte 

 Available quantity of root partition 
[[email protected] ~]# df -h | awk 'NR==2{print $4}'
45G

 3.1.15 awk Arithmetic

 awk Operation case of ：

[[email protected] ~]# awk 'BEGIN{x=10;print x}'  // If you don't use quotation marks awk Just output it as a variable , So there's no need to add $ 了 
10
[[email protected] ~]# awk 'BEGIN{x=10;print x+1}'   //BEGIN Before processing files , Therefore, not following the file name does not affect 
11
[[email protected] ~]# awk 'BEGIN{x=10;x++;print x}'
11
[[email protected] ~]# awk 'BEGIN{print x+1}'    // Do not specify the initial value , The initial value is 0, If it's a string , The default value is null 
1
[[email protected] ~]# awk 'BEGIN{print 2.5+3.5}'   // Decimals can also be calculated 
6
[[email protected] ~]# awk 'BEGIN{prawk 'BEGIN{print 2.5+3.5}'
1
[[email protected] ~]# awk 'BEGIN{print 3*4}'
12
[[email protected] ~]# awk 'BEGIN{print 3**2}'
9
[[email protected] ~]# awk 'BEGIN{print 2^3}'   //^ and ** It's all power operations 
8

[[email protected] ~]# awk 'BEGIN{print 1/2}' 
0.5

[[email protected] ~]# awk -F: '/root/' /etc/passwd      // If there is a specific number of columns printed later, it cannot be omitted print 了 
root:x:0:0:root:/root:/bin/bash
operator:x:11:0:operator:/root:/sbin/nologin


 Fuzzy matching , use ~ Means to contain ,!~ Does not include 
[[email protected] ~]# awk -F: '$1~/root/' /etc/passwd
root:x:0:0:root:/root:/bin/bash

[[email protected] ~]# awk -F: '$1~/ro/' /etc/passwd   // Fuzzy matching , As long as there is ro Just match 
root:x:0:0:root:/root:/bin/bash
chrony:x:993:988::/var/lib/chrony:/sbin/nologin
setroubleshoot:x:990:984::/var/lib/setroubleshoot:/sbin/nologin

[[email protected] ~]# awk -F: '$7!~/nologin$/{print $1,$7}' /etc/passwd

 

 

 3.1.16  Comparison of numeric value and string

About the comparison between numeric value and string
Comparative symbols ：== != <= >= <

 3.2.1 Print the fifth line

[[email protected] ~]# awk ‘NR== 5{print}’ /etc/passwd
lp：x:4:7:lp:/var/spool/lpd:/sbin/nologin
[[email protected] ~]# awk
‘NR==5’ /etc/passwd lp：x:4:7:lp:/var/spool/lpd:/sbin/nologin

 3.2.2 Print lines <5 The content of

[[email protected] ~]# awk  'NR<5' /etc/passwd
root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin

 3.2.3 The first column of exact matching and fuzzy matching is root Of

[[email protected] ~]# awk -F: '$3==0' /etc/passwd  // The third column is equal to 0 Of 
root:x:0:0:root:/root:/bin/bash

[[email protected] ~]# awk -F: '$1==root' /etc/passwd // The first column is root Of 
[[email protected] ~]# awk -F: '$1=="root"' /etc/passwd    // An exact match must be root
root:x:0:0:root:/root:/bin/bash

 3.2.4 The third column >=1000 Of

[[email protected] ~]# awk -F: '$3>=1000' /etc/passwd
nfsnobody:x:65534:65534:Anonymous NFS User:/var/lib/nfs:/sbin/nologin
shengjie:x:1000:1000:shengjie:/home/shengjie:/bin/bash

 3.2.5 Logical operations && ||

||: Or meaning

&&： And what it means

[[email protected] ~]# awk -F: '$3<10 || $3>=1000' /etc/passwd

||：

[[email protected] ~]# awk -F: '$3>10 && $3<1000' /etc/passwd

 &&：

[[email protected] ~]# awk -F: 'NR>4 && NR<10' /etc/passwd

3.3 Print a list of numbers

 Print 1-200 Between all can be 7 Divide and contain numbers 7 Integer number 
[[email protected] ~]# seq 200 | awk '$1%7==0 && $1~/7/'
7
70
77
147
175

 4. Other built-in variables

 Usage of other built-in variables FS（ Input ）、OFS、NR、FNR、RS、ORS
FS： Enter the delimiter of the field   Default is space 
OFS： Separator for output field   The default is also a space 
FNR： Number of records read from the file （ Line number ）, from 1 Start , The new file is redone 1 Start counting 
RS： Enter the line separator    The default is line break 
ORS： Output line separator    The default is line feed 

4.1 Define the field separator as colon before printing

[[email protected] ~]# awk 'BEGIN{FS=":"}{print $1}' hehe    
// Define the field separator as colon... Before printing 

 4.2//OFS Defines how the output is separated

[[email protected] ~]# awk 'BEGIN{FS=":";OFS="---"}{print $1,$2}' hehe

 //OFS Defines how the output is separated ,$1$2 The middle should be separated by commas , Because commas are mapped to by default OFS Variable , This variable defaults to a space
root---x
bin---x
daemon---x
adm---x
lp---x

 4.3 FNR The line number of is appended when there are multiple files

[[email protected] ~]# awk '{print FNR,$0}' /etc/resolv.conf /etc/hosts   // It can be seen that FNR The line number of is appended when there are multiple files 
1 
2 nameserver 114.114.114.114
3 search localdomain
1 127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
2 ::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
[[email protected] ~]# awk '{print NR,$0}' /etc/resolv.conf /etc/hosts
1 
2 nameserver 114.114.114.114
3 search localdomain
4 127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
5 ::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

 4.4RS: Specify what to use as the newline character , The colon specified here , What you specified must be the characters in the original text

[[email protected] ~]# awk 'BEGIN{RS=":"}{print $0}' /etc/passwd    //RS: Specify what to use as the newline character , The colon specified here , What you specified must be the characters in the original text 
root
x
0
0
root

4.5 Combine multiple lines into one line and output

[[email protected] ~]# awk 'BEGIN{ORS=" "}{print $0}' /etc/passwd  
 // Combine multiple lines into one line and output , When outputting, customize to separate each line with a space , The default is enter 

 4.6 awk Advanced usage of

 4.6.1 Define reference variables

[[email protected] ~]# a=100
[[email protected] ~]# awk -v b="$a" 'BEGIN{print b}'    // Set the system variables a, stay awk Assign a value to a variable b, And then call variables. b
100
[[email protected] ~]# awk 'BEGIN{print "'$a'"}'   // If you call directly, you need to use double quotation marks first and then single quotation marks 
100

 4.6.2 Call function getline, When reading a row of data, you do not get the current row, but the next row of the current row

[[email protected] ~]# df -h | awk 'BEGIN{getline}/root/{print $0}'

4.6.3 Show odd and even rows

[[email protected] ~]# seq 10 | awk '{print $0;getline}'   // Show odd rows 
1
3
5
7
9
[[email protected] ~]# seq 10 | awk '{getline;print $0}'   // Show even rows 
2
4
6
8
10

  5、 ... and .awk Of if sentence Single branch Multiple branches Double branch

if sentence ：awk Of if Statements are also divided into single branches 、 Two branches and many branches
Single branch is if(){}
The two branches are if(){}else{}
Multiple branches are if(){}else if(){}else{}

[[email protected] ~]# awk -F: '{if($3<10){print $0}}' /etc/passwd    // The third column is less than 10 Print entire line 
[[email protected] ~]# awk -F: '{if($3<10){print $3}else{print $1}}' /etc/passwd    // The third column is less than 10 Print the third column , Otherwise, print the first column 

  6、 ... and . summary

In a word, as the big brother of the three swordsmen ,AWK It's still very important , I hope you guys will take a serious look