当前位置:网站首页>Awk implements SQL like join operation
Awk implements SQL like join operation
2020-11-08 16:12:00 【xindoo】
awk、grep and sed go by the name of linux Three swordsmen , in fact grep and awk I also use it in my daily work (sed Use less ), Maybe some people are right about awk Less understanding , Let me give you a general introduction to . Many people think awk It's just a text processing tool , In fact, they use it the same way . But it's actually a language , Have mathematical operators 、 Process control statements , It even encapsulates many built-in variables and functions for text processing , This makes it powerful in text processing . If grep Can only do data filtering , that awk It can also process data 、 Analyze and even generate reports , After all, it's a complete programming language .
Because this article is not awk Getting started with , If you want to get started, I recommend Ruan Yifeng awk Introductory tutorial and Left ear mouse AWK A concise tutorial .
Back to my topic today , Today I'd like to show you something I use a lot awk Scene . Because we do back-end development , Often when doing data analysis, there will be such a problem ,1. Facing hundreds of thousands of data , We need to screen out hundreds and thousands of specific key The data of .2. For these millions of data , Directed against id Fields complement other fields .
There may be mastery of excel My classmates jumped out and said “ Is this ,so easy,vlookup That's it !” , in fact ,excel It does solve the problem , But it's a little heavy , Even sometimes we can't use it on the server excel. What other ways ? For the two scenes I'm talking about , Actually, think about it , Is it right? sql Two watches in Chinese join Can solve the problem . In fact, you don't really need to put files in the database , Only need to use awk One order can solve .
example
Let's take the question like , Suppose there are two files ,score.txt Save the student number + Performance data , Another one name.txt Save the student number + Name data , Now you want to know how many points everyone has baked .
score.txt
id score
1 87
2 67
3 68
4 75
5 90
6 100
7 0
name.txt
id name
1 Zhang San
2 Li Si
3 Wang Wu
4 Zhao er
5 Lennon
6 Big bear
You want a copy with a student number Name and grade data , It looks like this .
id score name
1 87 Zhang San
2 67 Li Si
3 68 Wang Wu
4 75 Zhao er
5 90 Lennon
6 100 Big bear
7 0
use awk How easy it is to generate such data ? Just one line of code , You can save name.txt and score.txt, And then execute the following command to try .
awk 'ARGV[1]==FILENAME {map[$1]=$2} ARGV[2]==FILENAME {print $0, map[$1]}' name.txt score.txt
It's simple name.txt and score.txt stay id Above right join.
Explain the code above ,ARGV and FILENAME yes awk Built in variables ,ARGV It's stored in awk List of accepted parameters , Like above ARGV[1] Namely "name.txt",ARGV[2] Namely "score.txt".awk It's line oriented , So for each row of data ARGV[1]==FILENAME {map[$1]=$2} ARGV[2]==FILENAME {print $0, map[$1]}
, Each row of data belongs to a file ,FILENAME Identify the file name to which the current line belongs , Like in brackets {}
Before ARGV[1]==FILENAME
You can think of it as in other grammars conditional , You can think of it as omitting if, But and if The function of is consistent .
So the meaning of the above code is If the current line is input name.txt Of , Just save the student number and name map in (awk You don't have to pre declare the variables in ). If the current line belongs to score.txt, From map Find out the name , And then output the data .
Conclusion
awk 'ARGV[1]==FILENAME {map[$1]=$2} ARGV[2]==FILENAME {print $0, map[$1]}' name.txt score.txt
For different data , Just adjust $ The specific value after , You can use different columns as key To do it join, It's not like sql Of join complex , But in linux It's very convenient on the server . On the Internet, I just realized right join, If in print $0, map[$1]
with if (length(map[$1]) > 0)
Can be realized inner join.left join You just need to change the file name .
I know that ,awk Realize the intersection of multiple files 、 The difference set and other operations are no exception . in addition Don't forget awk It's also a programming language , So it can be very complex , You can put the code in a file and use -f The parameters are set up , For example, my teacher used to use awk Do some simple statistical work , For example, calculate the mean value 、 The sum of the …… awk It can be called The back-end engineer is a sharp tool to improve efficiency .
This article is from https://blog.csdn.net/xindoo
版权声明
本文为[xindoo]所创,转载请带上原文链接,感谢
边栏推荐
- vim-配置教程+源码
- 搭载固态硬盘的服务器究竟比机械硬盘快多少
- SQL quick query
- B站stm32视频学习
- C + + things: from rice cookers to rockets, C + + is everywhere
- laravel8更新之速率限制改进
- I used Python to find out all the people who deleted my wechat and deleted them automatically
- 我用 Python 找出了删除我微信的所有人并将他们自动化删除了
- 漫画:寻找股票买入卖出的最佳时机(整合版)
- 我用 Python 找出了删除我微信的所有人并将他们自动化删除了
猜你喜欢
Improvement of maintenance mode of laravel8 update
模板引擎的整理归纳
刚刚好,才是最理想的状态
[开源] .Net 使用 ORM 访问 华为GaussDB数据库
I used Python to find out all the people who deleted my wechat and deleted them automatically
Liteos message queuing
我用 Python 找出了删除我微信的所有人并将他们自动化删除了
How to cooperate with people in software development? |Daily anecdotes
Flink的sink实战之一:初探
总结: 10月海外DeFi新项目,更多资管策略来了!
随机推荐
I used Python to find out all the people who deleted my wechat and deleted them automatically
2020-11-05
Golang ICMP Protocol detects viable hosts
wanxin finance
Using k3s to create local development cluster
机械硬盘随机IO慢的超乎你的想象
Improvement of maintenance mode of laravel8 update
Blockchain weekly: the development of digital currency is written into the 14th five year plan; Biden invited senior adviser of MIT digital currency program to join the presidential transition team; V
AI周报:允许“员工自愿降薪”;公司回应:员工内心高兴满意;虎牙HR将员工抬出公司;瑞典禁用华为中兴5G设备
SQL quick query
Travel notes of Suzhou
Learn to record and analyze
搭载固态硬盘的服务器究竟比机械硬盘快多少
C + + things: from rice cookers to rockets, C + + is everywhere
Stm32uberide download and install - GPIO basic configuration operation - debug (based on CMSIS DAP debug)
How to make a correct summary for 7 years?
刚刚好,才是最理想的状态
Welcome to offer, grade P7, face-to-face sharing, 10000 words long text to take you through the interview process
漫画:寻找股票买入卖出的最佳时机(整合版)
我们做了一个医疗版MNIST数据集,发现常见AutoML算法没那么好用