Flink practice tutorial: getting started 1- zero basic users realize simple Flink tasks

Oceanus brief introduction

Flow calculation Oceanus It is a powerful tool for real-time analysis of big data product ecosystem , Is based on Apache Flink Built with one-stop development 、 Seamless connection 、 Sub second delay 、 Low cost 、 Enterprise class real-time big data analysis platform with the characteristics of security and stability .Oceanus Provides a convenient console environment , It is convenient for users to write SQL Analysis statement 、ETL Job or upload and run custom JAR package , Support operation and maintenance management .

This article will show you how to use datagen and blackhole Connector randomly generates and stores data , To achieve one of the simplest Flink Mission .

Zero base user implementation is simple Flink Mission

One 、 Lead to

establish Oceanus colony

Activity purchase link 1 Yuan purchase Oceanus colony .

Get into Oceanus Console , Click on the left side of the 【 Cluster management 】, Click on the top left 【 Create clusters 】, For details, please refer to Oceanus Official documents Create an exclusive cluster .

Two 、 establish Oceanus Homework

1. establish Source

-  See  https://ci.apache.org/projects/flink/flink-docs-release-1.13/zh/dev/table/connectors/datagen.html
CREATE TABLE random_source ( 
  user_id INT,
  item_id INT,
  behavior VARCHAR
  ) WITH ( 
  'connector' = 'datagen',
  'rows-per-second' = '1',              --  The number of data generated per second 
  'fields.user_id.kind' = 'sequence',   --  Bounded sequence （ Automatically stop the output when finished ）
  'fields.user_id.start' = '1',         --  The starting value of the sequence 
  'fields.user_id.end' = '10000',       --  The end value of the sequence 
  'fields.item_id.kind' = 'random',     --  Unbounded random number 
  'fields.item_id.min' = '1',           --  The minimum of a random number 
  'fields.item_id.max' = '1000',        --  The maximum of random numbers 
  'fields.behavior.length' = '5'        --  The length of the random string 
);

2. establish Sink

--  See  https://ci.apache.org/projects/flink/flink-docs-release-1.13/zh/docs/connectors/table/blackhole/

CREATE TABLE blackhole_sink (
  user_id INT,
  item_id INT,
  behavior VARCHAR
) WITH ('connector' = 'blackhole');

3. Writing business SQL

INSERT INTO blackhole_sink
(
    SELECT user_id,
    item_id,
    behavior
    FROM random_source
);

4. Release operation

Click on the toolbar 【 Syntax check 】 Conduct SQL Syntax check , Click... After checking 【 preservation 】>【 Release the draft 】 Run job .

3、 ... and 、 summary

• Datagen Connector Connector is a tool for generating random data Connector, Generally used as a test .
• Sink To Blackhole The data will be discarded , The user cannot query the data , This connector is generally used for performance testing .

Reference reading

[1] data gen connector Reference link ：https://ci.apache.org/projects/flink/flink-docs-release-1.13/zh/docs/connectors/table/datagen/

[2] blackhole connecotr Reference link ：https://ci.apache.org/projects/flink/flink-docs-release-1.13/zh/docs/connectors/table/blackhole/