Installation and testing of pyflink

pyflink install

Installation premise ：python3.6-3.8

Reference resources ：Installation | Apache Flink

Python version (3.6, 3.7 or 3.8) is required for PyFlink. Please run the following command to make sure that it meets the requirements:

$ python --version

# the version printed here must be 3.6, 3.7 or 3.8

Here you can install python3 perhaps Anaconda3, Finally through python -V Command to view version information

Two installation methods ：

This installation is based on flink1.13.2 edition

If you have the Internet ： You can install directly through the command

python -m pip install apache-flink==1.13.2

Source compilation method

In addition you need Maven 3 and a JDK (Java Development Kit). Flink requires at least Java 8 to build.

maven choice 3.2.5 edition ,java Choose a higher one java8 edition

Reference resources ：Building Flink from Source | Apache Flink

Download the source code ：（ Here I downloaded it from other websites 1.13.2 Source code ）

git clone https://github.com/apache/flink.git

compile ： Errors may be reported during compilation , Just solve it concretely

mvn clean install -DskipTests 

#To speed up the build you can skip tests, QA plugins, and JavaDocs:

 perhaps ：mvn clean install -DskipTests -Dfast

Start processing after compilation pyflink Things about

install gcc【 Optional 】

yum install -y gcc gcc-c++

Installation dependency （flink-python Directory in flink Under the compilation directory ）

 python -m pip install -r flink-python/dev/dev-requirements.txt

Then go to flink The root directory of the source code and run this command to build and Of sdist Bao He wheel package ：apache-flink,apache-flink-libraries Of sdist The package can be Find below . It can be installed as follows ：apache-flink-libraries``./flink-python/apache-flink-libraries/dist/

cd flink-python; python setup.py sdist bdist_wheel; cd apache-flink-libraries; python setup.py sdist; cd ..;

If it is the intranet of the company , You need to configure pip Source , There are dependencies during installation , Because when I compile, I use a virtual machine , You can go online

python -m pip install apache-flink-libraries/dist/*.tar.gz

python -m pip install dist/*.tar.gz

adopt pip list | grep flink Command to view the installation effect

test

Local-SingleJVM Deployment mode

This pattern is mostly used in the development and testing phase , Make simple use of Python pyflink_job.py command ,PyFlink It will start one by default Local-SingleJVM Of Flink Environment to carry out the work , as follows ：

Write a script ：wordcount.py

from pyflink.table import DataTypes, TableEnvironment, EnvironmentSettings

from pyflink.table.descriptors import Schema, OldCsv, FileSystem

from pyflink.table.expressions import lit

settings = EnvironmentSettings.new_instance().in_batch_mode().use_blink_planner().build()

t_env = TableEnvironment.create(settings)

# write all the data to one file

t_env.get_config().get_configuration().set_string("parallelism.default", "1")

t_env.connect(FileSystem().path('/tmp/input')) \

    .with_format(OldCsv().field('word', DataTypes.STRING())) \

    .with_schema(Schema().field('word', DataTypes.STRING())) \

    .create_temporary_table('mySource')

t_env.connect(FileSystem().path('/tmp/output')) \

    .with_format(OldCsv().field_delimiter('\t') \

    .field('word', DataTypes.STRING()) \

    .field('count', DataTypes.BIGINT())) \

    .with_schema(Schema() \

    .field('word', DataTypes.STRING()) \

    .field('count', DataTypes.BIGINT())) \

    .create_temporary_table('mySink')

tab = t_env.from_path('mySource')

tab.group_by(tab.word).select(tab.word, lit(1).count).execute_insert('mySink').wait()

stay shell Command line execution ：

echo -e  "flink\npyflink\nflink" > /tmp/input

python wordcount.py

cat /tmp/output

Reference resources ：PyFlink Installation and use

Local-SingleNode Deployment mode

This mode is generally used for deployment in a stand-alone environment , Such as IoT In the device , We from 0 Start the deployment operation of this mode . We enter flink/build-target Catalog , Execute the following command :

 cd /root/flink-1.13.2/build-target/bin/

 ./start-cluster.sh

land http://ip:8081 see

Submit the assignment ：

/root/flink-1.13.2/build-target/bin/flink run -m localhost:8081 -py /root/wordcount.py

Reference resources ：PyFlink Multiple deployment modes of jobs

pyflink More related articles about the installation and testing of

my SQL Download and install , Environment configuration , And the solution of forgetting the password , as well as navicat for mysql download , install , Test connection
One . download Search on Baidu "mysql-5.6.24-winx64 download " Two . install Select installation path , My path “C:\Soft\mysql-5.6.24-winx64” 3、 ... and . Environment configuration Calculation ...
OpenCV2+ Introductory series （ One ）：OpenCV2.4.9 Installation and testing of
Let's assume that all the people who read this article have been right about OpenCV And machine vision and other basic concepts have a certain understanding , Therefore, this article will not focus on OpenCV Make any introduction , It's a direct introduction OpenCV2.4.9 Installation and testing of . In addition, this article only briefly introduces how to ...
Big data 3 -Apache ZooKeeper Standalone And copy mode installation and testing
Big data 3 -Apache ZooKeeper Standalone And copy mode installation and testing [TOC] Apache ZooKeeper Stand alone mode installation establish hadoop user & give sudo jurisdiction , ...
coreseek actual combat ( One )：windows Next coreseek Installation and testing of
coreseek actual combat ( One ):windows Next coreseek Installation and testing of About online coreseek stay windows There are many installation and use tutorials under , The government also has a detailed tutorial , Here I just follow the official tutorial ...
Window Next memcached Installation and testing steps
How to be in Window Next memcached Installation and testing steps Tools / raw material The computer memcached Software Method / step Package download download Memercached For Windows:http://downlo ...
【MySql】5.6.14 Installation and testing of version
current state :apache2.4.6 and php5.5.6 Installed successfully : mysql Installation and testing of : One . install mysql5.6.14, Reference resources http://wenku.baidu.com/link?url=_0jk ...
jdbc_odbc SQLserver Drive installation and testing
Yes 2 I was asked the same question twice , Although the blog Garden is .net Garden , I'd better share it .PS: What I'm doing now .net, I learned a little before java. I made a fool of myself . ------------------ Original mail -------------- ...
mosquitto stay Linux Deployment in environment / install / Use / test
mosquitto stay Linux Deployment in environment Read the source code for three or four days ,( Of course, I didn't take a good look ), Suddenly found out right mosquitto Source code has a little feeling , So on the fifth day, I decided to go to the hospital Linux Deploy... In the environment mosquitto. ...
http_load Installation and test parameter analysis - The pursuit of free programming - ITeye Technology website
http_load Installation and test parameter analysis - The pursuit of free programming - ITeye Technology website http_load -p 50 -s 120 urls
Docker Installation and testing of
1,Docker install Docker What is it , And its comparison with virtual machines , No introduction , There are many resources on the Internet to learn and understand . This article focuses on Docker Installation and testing of . Docker Installation , It is divided into offline installation and online installation ...

Random recommendation

springMVC Within the framework of —— General interface image upload interface
The image upload interface I want refers to the server after the image upload is completed , Return an accessible picture address . spring mvc It's very easy to upload pictures under the frame , as follows @RequestMapping(value="/upload ...
React A simple example of
First episode : Personal blog , to update & error correction & reply React At the heart of this is defining component classes , Components have three elements : state . Behavior . Interface . 1. Render state to interface : The state is determined by the state Attribute holds , Rendering from state to interface is done by components ...
Judge whether it contains Chinese , Include return true, Does not include return false
/** * function : Judge whether it contains Chinese , Include return true, Does not include return false */ function isChina(s) { var patrn = /[\u4E00-\u9FA5]|[\uFE30- ...
JQuery Upload plug-ins Uploadify API Detailed explanation
One . relevant key Value introduction uploader:uploadify.swf The relative path of the file , The swf The file is a file with text BROWSE The button , Click and fade out to open the file dialog box , The default value is :uploadify.swf. scrip ...
zoj 3823 Excavator Contest structure
Excavator Contest Time Limit: 1 Sec Memory Limit: 256 MB Topic linking http://acm.zju.edu.cn/onlinejudge/show ...
About wordpress Forget the password The way to retrieve the password
1. By directly modifying the encrypted character of the password in the database ( If wordpress Different versions of , So this method is not easy to implement ) 2. Use the way to retrieve the password : Find the password through email After the front-end login password error Will be displayed error ); Method 2 open WP ...
Network foundation IV DNS DHCP route FTP
The first 1 Chapter Network foundation 1.1 IP Address classification IP Type of address - Press IP Address value range Division IP Type of address - Press IP Address usage classification IP Type of address - Divided by network communication mode 1.2 Principle and process of LAN networking DHCP Principle process details ...
SQLServer And DEFAULT constraint
DEFAULT Constraint addition rules 1. If the default value constraint is defined in the table , When a user inserts a new data row , If the row does not specify data , Then the system assigns the default value to the column , If we don't set the default value , The system defaults to NULL. 2. If “ The default value is ” In the field ...
node.js Callback in
Synchronization and blocking : The two terms can be used interchangeably , It means that the execution of the code will stop before the function returns . If an operation is blocked , Then the script cannot continue , That means you have to wait . Asynchronous and non blocking : The two terms can be used interchangeably , Refers to callback based . Allow scripts to run in parallel ...
Allow Zero Length Allow empty string ACCESS
http://www.360doc.com/content/11/0118/20/991597_87447868.shtml https://microsoft.public.data.ado.nar ...