Flink cluster construction

Flink cluster setup

Local run mode

You can run it in IDEA, mainly for development

StandAlone Mode

Flink's own resource management framework does not need to depend on any other framework

1, upload and decompress

tar -zxvf flink-1.11.2-bin-scala_2.11.tgz -C /usr/local/soft/

2. Modify environment variables

vim /etc/profile# Configure flink's bin directory to the environment variable PATH

3. Modify the configuration file

cd /usr/local/soft/flink-1.11.2/confvim flink-conf.yaml# Change localhost to masterjobmanager.rpc.address: master# configure slave nodevim workersnode1node2# Modify the master addressvim mastersmaster:8081

4, sync to node1, node2

cd /usr/local/soft/scp -r flink-1.11.2/ node1:`pwd`scp -r flink-1.11.2/ node2:`pwd`

5. Start the cluster

# Only run on masterstart-cluster.sh

6. Access Flink's WEB UI

http://master:8081

7. Submit task

• Submit tasks through the flink command

  flink run -c com.shujia.core.Demo01WordCount Flink-1.0.jar

• Submit tasks through the web interface

• Submit tasks via RPC remote calls

8. Stop the cluster

stop-cluster.sh

On Yarn Mode

Mainstream operation mode, mainly used in production environment

yarn-session first starts a jobMansager in yarn, and all tasks share a jobmanager (submit tasks faster, share jobmanager between tasks, and affect each other)

Submit the task model directly, start a jobmanager for each task (each task is independent jobmanager, the task runs stably)

Preparation

• Configure environment variables, add HADOOP_CONF_DIR

  vim /etc/profileexport HADOOP_CONF_DIR=/usr/local/soft/hadoop-2.7.6/etc/hadoop/

• Add HADOOP dependencies

  cp flink-shaded-hadoop-2-uber-2.7.5-10.0.jar /usr/local/soft/flink-1.11.2/lib/

• Start Hadoop cluster

  # Execute on masterstart-all.sh

Yarn-Session Mode

It is equivalent to starting a JobManager in Yarn. When a task is submitted, it will apply for the TaskManager required for resource startup according to the parallelism of the task

• Start: yarn-session.sh -jm 1024m -tm 1024m
• Access: there is an address in the log after startup or jump through Yarn WEB UI
• Submit tasks:
  • Command: flink run -c com.shujia.core.Demo01WordCount Flink-1.0.jar
  • Submit via page
  • Submit via RPC

Direct On Yarn

It is equivalent to generating a JobManager each time a task is submitted

• Just start Yarn

• Flink tasks can only be submitted using commands

  flink run -m yarn-cluster -yjm 1024m -ytm 1024m -c com.shujia.core.Demo01WordCount Flink-1.0.jar

• Stop task

  • Cancel via the Cancel button on the JobManager interface

  • kill via Yarn's command

    yarn application -kill application_1654850164441_0002

• View logs

  yarn logs -applicationId application_1654850164441_0002