Flink cluster construction and enterprise level yarn cluster construction

Flink download ：

Apache Flink: download

According to the version you want to download , Download the specified version I downloaded 7.2hadoop edition

Unzip and start ：

Local boot

[[email protected] flinkhadoop1.7.2]# bin/start-scala-shell.sh local 

Submit one locally job

scala> benv.readTextFile("/home/admin/word.txt").flatMap(_.split(" ")).map((_,1)).groupBy(0).sum(1).print()

Stand alone cluster startup ：

[[email protected] flinkhadoop1.7.2]# bin/start-cluster.sh 
Starting cluster.
Starting standalonesession daemon on host node01.
Starting taskexecutor daemon on host node01.
[[email protected] flinkhadoop1.7.2]# jps
28417 StandaloneSessionClusterEntrypoint
29013 Jps
28858 TaskManagerRunner 

View the start cluster page

  Submit task to flink Single node cluster

[[email protected] flinkhadoop1.7.2]# bin/flink run /home/admin/flinkhadoop1.7.2/examples/batch/WordCount.jar --input /home/admin/word.txt  --output /home/admin/out

The front end views the uploaded tasks

Implementation plan ：

Output file display

[[email protected] flinkhadoop1.7.2]# cat /home/admin/out flink 1 hello 6 java 1 storm 1 stream 2 world 1

Stop the cluster ：

bin/stop-cluster.sh

http:// The server Ip:8081/ 

jobmanager.rpc.address: node01
jobmanager.rpc.port: 6123
jobmanager.heap.size: 1024
taskmanager.heap.size: 1024
taskmanager.numberOfTaskSlots: 2
taskmanager.memory.preallocate: false
parallelism.default: 1
jobmanager.web.port: 8081
taskmanager.tmp.dirs: /home/user/apps/flink/tmp
# Page submission 
web.submit.enable: true

[[email protected] flinkhadoop1.7.2]# vim conf/masters 
192.168.36.138:8081

[[email protected] flinkhadoop1.7.2]# vim conf/slaves 192.168.36.138 192.168.36.139 192.168.36.140

hadoop download  

https://archive.apache.org/dist/hadoop/core/hadoop-2.7.5/

hadoop Cluster deployment ：

Nanny level Hadoop Cluster deployment _ Wang, programmer java The blog of -CSDN Blog

Single deployment hadoop:

Single deployment hadoop2.7.3_ Wang, programmer java The blog of -CSDN Blog _hadoop2.7.3

hadoop visit ：

http://node01:50070/dfshealth.html#tab-overview

/usr/soft/hadoop-2.7.3/sbin/stop-dfs.sh
/usr/soft/hadoop-2.7.3/sbin/start-dfs.sh

hadoop create a file ： 
hdfs dfs -mkdir -p /word 
#1.  see hdfs File system directory file  
hdfs dfs -ls /word
 #2. Upload ：
hdfs dfs -put  Local file directory  HDFS File directory  
hdfs dfs -put /usr/apps/word/words.txt /word
#3. Delete file   hdfs dfs -rm -r HDFS File path  
hdfs dfs -rm -r /word

flink Cluster deployment , The same routine ; Cluster deployment and enterprise level Hadoop colony yarn Refer to the following address for deployment https://blog.csdn.net/weixin_44385486/article/details/124197370?ops_request_misc=%257B%2522request%255Fid%2522%253A%2522165552127416782388044840%2522%252C%2522scm%2522%253A%252220140713.130102334.pc%255Fblog.%2522%257D&request_id=165552127416782388044840&biz_id=0&utm_medium=distribute.pc_search_result.none-task-blog-2~blog~first_rank_ecpm_v1~rank_v31_ecpm-3-124197370-null-null.nonecase&utm_term=flink&spm=1018.2226.3001.4450https://blog.csdn.net/weixin_44385486/article/details/124197370?ops_request_misc=%257B%2522request%255Fid%2522%253A%2522165552127416782388044840%2522%252C%2522scm%2522%253A%252220140713.130102334.pc%255Fblog.%2522%257D&request_id=165552127416782388044840&biz_id=0&utm_medium=distribute.pc_search_result.none-task-blog-2~blog~first_rank_ecpm_v1~rank_v31_ecpm-3-124197370-null-null.nonecase&utm_term=flink&spm=1018.2226.3001.4450

In the process of cluster building , Possible problems ：

• Port occupied , We need to manually kill the program that occupies the port ;
• Directory not found or file not found , We are flink-conf.yaml I've configured io.tmp.dirs , This directory needs to be created manually

 flink -hadoop Highly available clusters and yarn need Corresponding flink-hadoop rely on , Download address

flink-hadoop rely on

command ：

./bin/flink run -m yarn-cluster -yid application_xxxx ./examples/batch/WordCount.jar
yarn Run task on

./bin/flink run -yjm 1024m -ytm 4096m -ys 2  ./examples/batch/WordCount.jar

Start cluster

bin/start-cluster.sh

Stop the cluster

bin/stop-cluster.sh

The attachment FLink Technical data can be referred to ：

FLink technical knowledge