awk、grep and sed go by the name of linux Three swordsmen , in fact grep and awk I also use it in my daily work (sed Use less ), Maybe some people are right about awk Less understanding , Let me give you a general introduction to . Many people think awk It's just a text processing tool , In fact, they use it the same way . But it's actually a language , Have mathematical operators 、 Process control statements , It even encapsulates many built-in variables and functions for text processing , This makes it powerful in text processing . If grep Can only do data filtering , that awk It can also process data 、 Analyze and even generate reports , After all, it's a complete programming language .
Because this article is not awk Getting started with , If you want to get started, I recommend Ruan Yifeng awk Introductory tutorial and Left ear mouse AWK A concise tutorial .
Back to my topic today , Today I'd like to show you something I use a lot awk Scene . Because we do back-end development , Often when doing data analysis, there will be such a problem ,1. Facing hundreds of thousands of data , We need to screen out hundreds and thousands of specific key The data of .2. For these millions of data , Directed against id Fields complement other fields .
There may be mastery of excel My classmates jumped out and said “ Is this ,so easy,vlookup That's it ！” , in fact ,excel It does solve the problem , But it's a little heavy , Even sometimes we can't use it on the server excel. What other ways ？ For the two scenes I'm talking about , Actually, think about it , Is it right? sql Two watches in Chinese join Can solve the problem . In fact, you don't really need to put files in the database , Only need to use awk One order can solve .
example
Let's take the question like , Suppose there are two files ,score.txt Save the student number + Performance data , Another one name.txt Save the student number + Name data , Now you want to know how many points everyone has baked .
score.txt
id score
1 87
2 67
3 68
4 75
5 90
6 100
7 0
name.txt
id name
1 Zhang San
2 Li Si
3 Wang Wu
4 Zhao er
5 Lennon
6 Big bear
You want a copy with a student number Name and grade data , It looks like this .
id score name
1 87 Zhang San
2 67 Li Si
3 68 Wang Wu
4 75 Zhao er
5 90 Lennon
6 100 Big bear
7 0
use awk How easy it is to generate such data ？ Just one line of code , You can save name.txt and score.txt, And then execute the following command to try .
awk 'ARGV[1]==FILENAME {map[$1]=$2} ARGV[2]==FILENAME {print $0, map[$1]}' name.txt score.txt
It's simple name.txt and score.txt stay id Above right join.
Explain the code above ,ARGV and FILENAME yes awk Built in variables ,ARGV It's stored in awk List of accepted parameters , Like above ARGV[1] Namely "name.txt",ARGV[2] Namely "score.txt".awk It's line oriented , So for each row of data ARGV[1]==FILENAME {map[$1]=$2} ARGV[2]==FILENAME {print $0, map[$1]}, Each row of data belongs to a file ,FILENAME Identify the file name to which the current line belongs , Like in brackets {} Before ARGV[1]==FILENAME You can think of it as in other grammars conditional , You can think of it as omitting if, But and if The function of is consistent .
So the meaning of the above code is If the current line is input name.txt Of , Just save the student number and name map in (awk You don't have to pre declare the variables in ). If the current line belongs to score.txt, From map Find out the name , And then output the data .
Conclusion
awk 'ARGV[1]==FILENAME {map[$1]=$2} ARGV[2]==FILENAME {print $0, map[$1]}' name.txt score.txt
For different data , Just adjust $ The specific value after , You can use different columns as key To do it join, It's not like sql Of join complex , But in linux It's very convenient on the server . On the Internet, I just realized right join, If in print $0, map[$1] with if (length(map[$1]) > 0) Can be realized inner join.left join You just need to change the file name .
I know that ,awk Realize the intersection of multiple files 、 The difference set and other operations are no exception . in addition Don't forget awk It's also a programming language , So it can be very complex , You can put the code in a file and use -f The parameters are set up , For example, my teacher used to use awk Do some simple statistical work , For example, calculate the mean value 、 The sum of the …… awk It can be called The back-end engineer is a sharp tool to improve efficiency .
This article is from https://blog.csdn.net/xindoo
