Flink 1.15.1 Cluster Construction (StandaloneSession)

I. Preparation

1. Flink version

Apache Flink 1.15.1 for Scala 2.12

2. System platform

linux debian11.4_64.

3. Node

• 192.168.1.25 (flink-master, flink-worker-1)
• 192.168.1.26 (flink-worker-2)
• 192.168.1.27 (flink-worker-3)

4. Configuration

(1) Local DNS settings

Add the following to the /etc/hosts file of the three nodes:

192.168.1.25 flink-master192.168.1.25 flink-worker-1192.168.1.26 flink-worker-2192.168.1.27 flink-worker-3

(2) ssh password-free login settings

Flink standalone clusters run sshd to execute scripts for managing Flink components.

1) Generate a secret key

Execute the following commands on the three nodes respectively:

ssh-keygen

2) Install the public key

Install the public key of the current node on other nodes (including the current node), that is, add the content of the ~/.ssh /id_rsa.pub file of this node to the ~/.ssh/authorized_keys file of other cluster nodes, thenOther cluster nodes can log in to this node without password.

Execute the following three commands on the three nodes respectively:

ssh-copy-id flink-worker-1ssh-copy-id flink-worker-2ssh-copy-id flink-worker-3

Second, build a cluster

1. Installation path

Unzip the downloaded Flink binary package and copy it to the /opt/flink-1.15.1 path.The three nodes are the same.

2. Configure the cluster node list

It can be configured on the master node.

1) Primary node (jobmanager)

Modify the /opt/flink-1.15.1/conf/masters file and add the master node address as follows:

flink-master:8081

2) Work node (taskmanager)

Modify the /opt/flink-1.15.1/conf/workers file and add the worker node address as follows:

flink-worker-1flink-worker-2flink-worker-3

By configuring masters and workers, the start-cluster.sh and stop-cluster.sh scripts can be used to uniformly manage (start, shut down, etc.) all cluster nodes, which is convenient and fast.

We can also not configure masters, workers, and manage sets without using start-cluster.sh and stop-cluster.sh scripts, directly by executing " jobmanager.sh start | stop ", " taskmanager.sh start | stop "To start the master node and worker nodes (clusters started through start-cluster.sh, you can also dynamically add or remove worker nodes through taskmanager.sh).

3. Cluster parameter configuration

Modify the /opt/flink-1.15.1/conf/flink-conf.yaml file of the three nodes respectively. The key content is as follows (other parameters are set as appropriate):

# master addressjobmanager.rpc.address: flink-master# master portjobmanager.rpc.port: 6123# master address binding settings (master node parameters)jobmanager.bind-host: 0.0.0.0# worker address binding settingstaskmanager.bind-host: 0.0.0.0# worker address (note: the hosts of the three worker nodes are different)taskmanager.host: flink-worker-1# worker slot number settingtaskmanager.numberOfTaskSlots: 2# default parallelismparallelism.default: 2# WEB UI port (master node parameter)rest.port: 8081# WEB UI management addressrest.address: flink-master# WEB UI address binding settings, if you want external access, you can set a specific IP, or directly set it to "0.0.0.0" (master node parameter)rest.bind-address: 0.0.0.0# Job file directory (master node parameter)web.upload.dir: /opt/flink-1.15.1/usrlib# IO temporary directory, default: /tmpio.tmp.dirs: /opt/flink-1.15.1/tmp# Cluster node process ID storage directory, default: /tmpenv.pid.dir: /opt/flink-1.15.1/pids

Note:

The "taskmanager.host" parameters of the three worker nodes are different and can be changed to their respective addresses.The number of slots and parallelism of each worker node can also be set according to the conditions of the node itself.

4. Start the cluster

On the master node, the cluster can be started through the /opt/flink-1.15.1/bin/start-cluster.sh script (standalonesession mode).Running this script automatically starts the jobmanager node and the other three taskmanagers in the cluster (no need to start taskmanager on other nodes).

Startup process:

Admin interface (overview):

Management interface (jobmanager):

Management interface (taskmanager):

5. Close the cluster

On the master node, run the /opt/flink-1.15.1/bin/stop-cluster.sh script, which will stop the jobmanager and all taskmanagers at the same time.

Second, execute JOB

Upload task JAR:

Task Status: