当前位置：网站首页>CDH cluster integration external Flink (improved version - keep pace with the times)

CDH cluster integration external Flink (improved version - keep pace with the times)

2022-07-27 04:53:00 【I want to express myself to the girl I like with code】

Preface

Refer to the article , Given cdh The compiled flink, It's off the shelf , So we have to find a way , compile .

It's not terrible to refer to other people's articles , What I'm afraid of is , Not making progress , We can refer to others' articles , And for deficiencies , Improvement , Will be excellent , To study , In this way, the technical articles and communities of our country and nation , To make progress .

Maybe one day I won't be a programmer , But the technology has been updated , At that time , I hope too , My article can be taken away by others , Improvement , Benefit the open source community .

therefore , This article is based on reference articles , Improved version

need

cdh colony （ Premise ）--- I've set it up

flink Of -parcel Package production

Flink Parcel package

1） Download the production package

git clone https://github.com/pkeropen/flink-parcel.git

open git bash, After successful cloning , A folder will be generated in the current file

Upload folder to centos System , Because of the need to use sh,( You go straight from centos On the system git clone It's fine too )

2） Modify the configuration file

Enter the production package just downloaded , modify flink-parcel.properties file

The configuration is as follows

flink Slow download , So I downloaded it in advance and put it into the local httpd Server

I am here apache Downloaded from the official website , Slow download

https://dlcdn.apache.org/flink/flink-1.12.7/flink-1.12.7-bin-scala_2.12.tgz

So it became a domestic image of Huateng , Speed up

https://mirrors.cloud.tencent.com/apache/flink/flink-1.12.7/flink-1.12.7-bin-scala_2.12.tgz

Put the downloaded file in your local httpd Catalog ,/var/www/html/ Next

#flink Download address
FLINK_URL=http://192.168.30.101:httpd port /flink File path
#flink Version number
FLINK_VERSION=1.12.7
# Extended version number
EXTENS_VERSION=BIN-SCALA_2.12
# Operating system version , With centos For example
OS_VERSION=7
#CDH Small version , I am a 6.3.2 The cluster of , So I set the maximum value to 6.3.3
CDH_MIN_FULL=5.2
CDH_MAX_FULL=6.3.3
#CDH The big version
CDH_MIN=5
CDH_MAX=6

take build.sh and flink-parcel.properties Document format Convert to UNIX Format

(1) Upload the folder to the service

(2)linux Installation on git

sudo yum install -y git

(3)linux install maven

maven download , Although only 8MB, But the official website downloads slowly

I still downloaded it

decompression

tar -zxvf apache-maven-3.8.6-bin.tar.gz

This is my unzipped directory /software/apache-maven-3.8.6

Environment variable configuration （ Plus this 2 sentence ）

vi /etc/profile

export MAVEN_HOME=/software/apache-maven-3.8.6
export PATH=$PATH:$MAVEN_HOME/bin

to update

source /etc/profile

test

mvn- v

Because the great God wrote Parcel Of build.sh Scripts need to rely on maven and git as well as sh also java

3)build.sh Use

parcel package

Use the... Of the current directory build.sh Script execution

./build.sh  parcel

After success, it will be generated in the current directory

structure flink-yarn csd package

./build.sh csd_on_yarn

Successful generation

4） pack

tar -cvf ./FLINK-1.12.7-BIN-SCALA_2.12.tar ./FLINK-1.12.7-BIN-SCALA_2.12_build/

Generate

On top 1 Level directory , Generate target package

Put this 2 A package ,copy In the past

（1） take FLINK-1.12.7-BIN-SCALA_2.11_build Put the files in /opt/cloudera/parcel-repo
（2） cp FLINK_ON_YARN-1.12.7.jar /opt/cloudera/csd/

Click Assign first , restart server,

restart server

systemctl restart cloudera-scm-server

then Activate at point , Originally, I took a screenshot of myself ,csdn Delete it for me ,ctrl+z Really vomit blood , Your design , Go back to the last auto save point directly , Instead of deleting the data I just entered .