当前位置:网站首页>Cartoon: what is MapReduce?
Cartoon: what is MapReduce?
2022-07-05 16:09:00 【Small ash】
————— the second day —————
————————————
What is? MapReduce?
MapReduce It's a programming model , The theory comes from Google Three papers published by the company (MapReduce,BigTable,GFS) One of , It is mainly used in parallel computing of massive data .
MapReduce Can be divided into Map and Reduce Two parts of understanding .
1.Map: Mapping process , Put a set of data in a certain way Map Functions map to new data .
2.Reduce: Reduction process , Several groups of mapping results are summarized and output .
Let's look at a practical chestnut , How to efficiently count the number of people with all surnames in the country ?
We can use MapReduce Thought , Do parallel mapping for the population of each province , Some local results are obtained , And then sort out and summarize these local results :
What does this picture mean ? Let's explain the steps :
1.Map:
In provinces , Multiple threads read the population data of different provinces in parallel , Each record generates a Key-Value Key value pair . Here's just simplified data .
2.Shuffle
Shuffle This concept has not been mentioned before , Its Chinese meaning is “ Shuffle ”.Shuffle The process is to sort the data maps 、 grouping 、 Copy .
3.Reduce
Results grouped before execution , And summarize and output .
It should be noted that , Described here Shuffle It's just an abstract concept , In the course of actual implementation Shuffle It's divided into two parts , Part of it is Map In the task , Part of it is Reduce In the task .
Hadoop How to achieve MapReduce?
Hadoop yes Apache A distributed system framework developed by the foundation , Contains multiple components , Its core is HDFS and MapReduce.
For reasons of length , The text won't be right Hadoop Make a complete introduction , Just a brief introduction to Haddoop How to achieve MapReduce.
Here is the graph Hadoop The framework performs a MapReduce Job The whole process :
There are several entities that need to be explained :
HDFS:
Hadoop Distributed file system , by MapReduce Provide data sources and Job Information storage .
Client Node:
perform MapReduce Process of procedure , To submit MapReduce Job.
JobTracker Node:
Put the whole Job Split into several Task, Responsible for dispatching and coordinating all Task, amount to Master Role .
TaskTracker Node:
Responsible for the execution by JobTracker Assigned Task, amount to Worker Role . Of these Task It is divided into MapTask and ReduceTask.
Last , I wish you guys who aspire to be big data engineers , And all the readers of Xiaohui , Achieve your dream in the new year !
—————END—————
边栏推荐
- 程序员如何提升自己的格局?
- Analytic hierarchy process of mathematical modeling (including Matlab code)
- Memo 00
- 定义严苛标准,英特尔Evo 3.0正在加速PC产业升级
- Example project: simple hexapod Walker
- Li Kou today's question -729 My schedule I
- 一键安装脚本实现快速部署GrayLog Server 4.2.10单机版
- Summary of the third class
- Information collection of penetration test
- ES6深入—ES6 Class 类
猜你喜欢
Arduino控制微小的六足3D打印机器人
Coding devsecops helps financial enterprises run out of digital acceleration
21. [STM32] I don't understand the I2C protocol. Dig deep into the sequence diagram to help you write the underlying driver
sql中set标签的使用
SQL injection sqllabs (basic challenges) 11-20
定义严苛标准,英特尔Evo 3.0正在加速PC产业升级
Background system sending verification code function
vulnhub-FirstBlood
ES6 drill down - Async functions and symbol types
Data communication foundation - dynamic routing protocol rip
随机推荐
Use of set tag in SQL
Data communication foundation - routing communication between VLANs
效果编辑器新版上线!3D渲染、加标注、设置动画,这次一个编辑器就够了
21. [STM32] I don't understand the I2C protocol. Dig deep into the sequence diagram to help you write the underlying driver
漫画:什么是MapReduce?
ES6 deep - ES6 class class
Relationship between objects and classes
Verilog realizes the calculation of the maximum common divisor and the minimum common multiple
ES6 drill down - ES6 generator function
[graduation season] as a sophomore majoring in planning, I have something to say
事务回滚异常
List uses stream flow to add according to the number of certain attributes of the element
一文搞定vscode编写go程序
Research and practice of super-resolution technology in the field of real-time audio and video
Lesson 4 knowledge summary
Vulnhub-Moneybox
CODING DevSecOps 助力金融企业跑出数字加速度
Background system sending verification code function
Go language programming specification combing summary
具有倍数关系的时钟切换