当前位置:网站首页>Spark calculation operator and some small details in liunx
Spark calculation operator and some small details in liunx
2022-07-06 17:39:00 【Bald Second Senior brother】
Spark -map operator
map operator :
object Spark01_Oper { def main(args: Array[String]): Unit = { val conf = new SparkConf().setMaster("local[*]").setAppName("Value") val cs = new SparkContext(conf) val make = cs.makeRDD(1 to 10) //map operator val mapRdd = make.map(x => x * 2) mapRdd.collect().foreach(println) } }
map Operators are used to calculate the data in all incoming partitions one by one .
mapPartRdd operator
object Spark02_OPer { def main(args: Array[String]): Unit = { val conf = new SparkConf().setMaster("local[*]").setAppName("mapPart") val sc = new SparkContext(conf) //map operator val list = sc.makeRDD(1 to 10) val mapPartRdd = list.mapPartitions(datas => {datas.map(data => data*2)}) mapPartRdd.collect().foreach(println) } }
mapPartRdd operator Be similar to map But it calculates data by partition , The output value of his calculation is a list
mapPartitionsWithIndex operator
object Spark03_OPer {
def main(args: Array[String]): Unit = {
val conf = new SparkConf().setMaster("local[*]").setAppName("With")
val sc = new SparkContext(conf)
val list = sc.makeRDD(1 to 10,2)
val indexRDD = list.mapPartitionsWithIndex {
case (num, datas) => {
datas.map((_," Zone number :"+num))
}
}
indexRDD.collect().foreach(println)
}
}
mapPartitionsWithIndex The operator is similar to mapPartiyions But in func There will be an index value representing the partition , therefore func There will be one more function similar to Int.
spark Possible problems :
Because every time the calculation data will produce new data, but it will not be deleted , Accumulating all the time will cause memory overflow (OOM)
Driver And Executor The difference between
Driver:
Driver Just create Spark The classes of context objects can be said to be Driver,Driver yes Spark in Application That is, the code release program , It can be understood that it is written for us spark The main program of the code , Secondly, he is also responsible for Executor To allocate tasks ,Driver There can only be one
Executor:
Executor yes Spark In charge of resource calculation , He can exist in multiple .
difference :
Drvier Like a boss , and Executor yes Driver The hands of ,Driver Be responsible for assigning tasks to Executor To execute .
Linux Pick up
linux Method of switching on and off
1. To turn it off : shutdown -h restart :shutdown -r
2. To turn it off : inti -0 restart : init -6
3. To turn it off : poweroff restart :reboot
service And systemctl The difference between
service:
You can start 、 stop it 、 Restart and shut down system services , It can also display the current status of all system services ,service The function of the command is to /etc/init.d Find the corresponding service under the directory , Open and close
systemctl:
It's a systemd Tools , Mainly responsible for control systemd System and service manager , yes service and chkconfig The combination of orders
Operation of network equipment :
Environment variable loading order
边栏推荐
- The art of Engineering (2): the transformation from general type to specific type needs to be tested for legitimacy
- 自动化运维利器-Ansible-Playbook
- Chrome prompts the solution of "your company management" (the startup page is bound to the company's official website and cannot be modified)
- 分布式(一致性协议)之领导人选举( DotNext.Net.Cluster 实现Raft 选举 )
- pip install pyodbc : ERROR: Command errored out with exit status 1
- MySQL报错解决
- Solr appears write Lock, solrexception: could not get leader props in the log
- JVM garbage collector part 1
- CTF逆向入门题——掷骰子
- CentOS7上Redis安装
猜你喜欢
案例:检查空字段【注解+反射+自定义异常】
连接局域网MySql
Selenium test of automatic answer runs directly in the browser, just like real users.
学 SQL 必须了解的 10 个高级概念
List集合数据移除(List.subList.clear)
Final review of information and network security (full version)
Huawei certified cloud computing hica
Display picture of DataGridView cell in C WinForm
复盘网鼎杯Re-Signal Writeup
03 products and promotion developed by individuals - plan service configurator v3.0
随机推荐
06个人研发的产品及推广-代码统计工具
C#WinForm中的dataGridView滚动条定位
Flink parsing (VII): time window
[mmdetection] solves the installation problem
C WinForm series button easy to use
Yarn: unable to load file d:\programfiles\nodejs\yarn PS1, because running scripts is prohibited on this system
mysql高級(索引,視圖,存儲過程,函數,修改密碼)
Automatic operation and maintenance sharp weapon ansible Foundation
微信防撤回是怎么实现的?
Xin'an Second Edition: Chapter 25 mobile application security requirements analysis and security protection engineering learning notes
虚拟机启动提示Probing EDD (edd=off to disable)错误
Akamai浅谈风控原理与解决方案
The solution to the left-right sliding conflict caused by nesting Baidu MapView in the fragment of viewpager
Xin'an Second Edition: Chapter 23 cloud computing security requirements analysis and security protection engineering learning notes
Akamai 反混淆篇
Based on infragistics Document. Excel export table class
Interpretation of Flink source code (I): Interpretation of streamgraph source code
JVM garbage collector part 1
Application service configurator (regular, database backup, file backup, remote backup)
华为认证云计算HICA