当前位置:网站首页>Basic configuration and use of spark
Basic configuration and use of spark
2022-07-06 17:39:00 【Bald Second Senior brother】
Catalog
Content :
spark Configuration of the three modes and spark The basic use method
Spark Three models of
Local Pattern ( Local mode )
local Set up Master Methods :local( Default to a thread ),local[k]( Specified number of threads ),local[*]( Most used cpu Set thread ); The thread executing is Worker
To configure :
local There is no need to modify the configuration file ,Spark After installation, you can use it directly local Model to calculate and analyze data
Standalone Pattern
Operation mode and hadoop Of resourcemanage Very similar .
The configuration file
Modify this file , If you don't change the name of his reference document
Modify the content :
Add the name of the host , Follow hadoop The same is for group and cluster services
This file also needs to be modified , If nothing, change the reference file
Configure the content
This is designated to run on that machine master, And specify the port number
The next step is to send the configuration file to slaves Each host in
This file modification specifies java The running address of
History server
To add a historical server, you need to modify this configuration file and add the following contents
Be careful : Take out the notes
As above, the configuration file needs to be modified
stay hdps Created in directory file , And start up hdfs
HA
Modify the configuration file
Add content
Appoint zookeeper Location of , And hdfs At the same time, comment out
To avoid conflict
Then distribute the configuration file
Yarn Pattern
effect : You don't need to build Spark Cluster of
The configuration file :
1.
modify hadoop in yarn Configuration file for
Modify the content :
The function of this configuration is to turn off operations with excessive memory , Otherwise, when the computing memory exceeds a certain limit spark It will turn off automatically
2. modify
Modify the content :
Be careful
Before using different modes, you should comment or delete the contents of other modes in the configuration file .
Use of official cases
Pay attention to each different mode ,master The content of is different , There will be some differences in other places
api Use
object WordCount { def main(args: Array[String]): Unit = { //WordCount Development //local Pattern // establish sparkConf object // Set up spark Deployment environment for val config = new SparkConf().setMaster("local[*]").setAppName("WordCount") // establish spark Context object val sc=new SparkContext(config) // Read the file , Read line by line ( Local search file:///xxxxx) val lines = sc.textFile("file:///opt/module/ha/spark/in") // Decompose the data into words one by one val words = lines.flatMap(_.split(" ")) // Transformation structure val wordToOne = words.map((_, 1)) // Group aggregation val wordToSum = wordToOne.reduceByKey(_ + _) // Printout println(wordToSum.collect()) } }
summary :
Today I learned how to spark At the same time, the official use of spark Three modes of and their configuration files ,spark The configuration file of is relatively simple, but it is inconvenient that it cannot coexist , At the same time, the use method of his official operation is too complex to be easy to remember , It is estimated that you still need to check the document for later use . Next is in java It is also cumbersome when used on , But it is much simpler than the command line
边栏推荐
- Flink analysis (II): analysis of backpressure mechanism
- Interpretation of Flink source code (I): Interpretation of streamgraph source code
- [introduction to MySQL] third, common data types in MySQL
- C#WinForm中的dataGridView滚动条定位
- C version selenium operation chrome full screen mode display (F11)
- List set data removal (list.sublist.clear)
- Final review of information and network security (full version)
- 微信防撤回是怎么实现的?
- Akamai anti confusion
- 当前系统缺少NTFS格式转换器(convert.exe)
猜你喜欢
Grafana 9 正式发布,更易用,更酷炫了!
Integrated development management platform
[rapid environment construction] openharmony 10 minute tutorial (cub pie)
Uipath browser performs actions in the new tab
Wu Jun's trilogy insight (V) refusing fake workers
Flink analysis (I): basic concept analysis
Junit单元测试
复盘网鼎杯Re-Signal Writeup
Yarn: unable to load file d:\programfiles\nodejs\yarn PS1, because running scripts is prohibited on this system
C WinForm series button easy to use
随机推荐
信息与网络安全期末复习(完整版)
TCP连接不止用TCP协议沟通
07个人研发的产品及推广-人力资源信息管理系统
【Elastic】Elastic缺少xpack无法创建模板 unknown setting index.lifecycle.name index.lifecycle.rollover_alias
【MySQL入门】第三话 · MySQL中常见的数据类型
C#版Selenium操作Chrome全屏模式显示(F11)
网络分层概念及基本知识
C# WinForm中DataGridView单元格显示图片
[CISCN 2021 华南赛区]rsa Writeup
[reverse] repair IAT and close ASLR after shelling
关于Selenium启动Chrome浏览器闪退问题
基于LNMP部署flask项目
Redis quick start
Wu Jun's trilogy experience (VII) the essence of Commerce
C# WinForm系列-Button简单使用
Junit单元测试
MySQL advanced (index, view, stored procedure, function, password modification)
The art of Engineering (2): the transformation from general type to specific type needs to be tested for legitimacy
【ASM】字节码操作 ClassWriter 类介绍与使用
[rapid environment construction] openharmony 10 minute tutorial (cub pie)