Play with the cluster configuration center and learn about the Taier console

Original address ： Play with the cluster configuration center , One article for you to understand Taier Console DTMO 03 Live review of this period （ Contains video + Courseware ）

Course video review ：Taier Basic course （ Two ）：Taier1.1 New features & Console Introduction _ Bili, Bili _bilibili

Open source project communication group ：30537511（ nailing ）

Open source project address ：GitHub - DTStack/Taier: Big data platform - Distributed task scheduling system

Last issue , The teacher introduced us to Taier To get started , In this issue, teacher Yuebai will focus on Taier The console will introduce you . The console is Taier Medium Cluster configuration center , Through the console Taier It can realize the docking of different cluster environments , Configure different types of components under each cluster , Then develop the corresponding tasks

Through the console ,Taier Realized Multi cluster 、 multi-tenancy Characteristics of .

Taier Introduction to console functions

Get into Taier after , First, you need to configure the cluster on the console , After configuring the cluster , You need to bind the tenant to the cluster to carry out task development . The functions of the console mainly include the following three modules ：

Multi cluster management

Multiple sets of cluster environments can be configured on the console , Different computing components can also be configured under the cluster through different computing components , To achieve the submission of different types of tasks .

This function functions as a configuration center , Only the corresponding cluster information can be configured into the console cluster , Only when the task is submitted to the cluster environment .

Resource management

The function of resource management display is to display the memory of cluster computing nodes 、 Audit usage and of tenants and clusters Binding relationship , Help users better determine the resource configuration and usage .

This feature is an effective help Taier Achieve tenant isolation , stay Taier The beginning of the design of , One of the first concepts defined is multi tenant and multi cluster isolation , This is in Taier in , It is realized through the mutual binding of tenants and clusters .

Queue management

Queue management is responsible for coordinating the task status of different nodes in the cluster , The user can operate it manually .

Its main function is to submit parameters , The parameter information submitted by the task is obtained and built from the corresponding components of the console .

Taier Console design

After explaining the function of the console , Next, let's share the design idea of the console ,Taier The design of the console is mainly introduced around the following points ：

Componentization

1、 Component structure logic

Because the information of different environments is inconsistent , Daily task information processing needs to configure multiple tasks colony （Cluster） To distinguish the source of information . The components under each cluster can be divided into... According to their functions public Components 、 resources Dispatch Components 、 Storage Components 、 Calculation Component these four functional components , Under the same functional component, more detailed distinction can be made according to different types , The design of the architecture is realized layer by layer from the whole to the part .

The main functions of the four functional components are ：

• Common components （Common Part）： Common component common implementation logic

• Resource scheduling component （Resource Part）： General implementation logic of resource scheduling component  

• Storage component （Store Part）： Common implementation logic of storage components  

•   Computing components （Compute Part）： General implementation logic of computing components , Include ：

DependComputePart： Common implementation logic of computing components that depend on resource components and storage components, such as ：Flink、Spark And other computing components
SingleComputePart： Computing components that do not rely on any other components, such as ：OceanBase、TiDB And other computing components

2、 Component implementation logic

The implementation logic of component includes the following aspects ：

• The corresponding unique type enumeration value of the component  

• The specific version of the component

• Template parameters for components

• Component correspondence plugin name

• The resource component on which the component depends

• Component additional version parameters

pluggable

How components correspond to a plug-in version , In version Apache Hadoop2 Take component plug-ins as an example , Specific command operations can be viewed in course playback , The plug-in files can be found in the following folder ：

Taier Console planning

simple

Taier The core idea of design has always been 「 Light and easy to use 」. To achieve this goal , We will take a series of measures in the future , For example, optimize the correlation between computing components and tasks , Make custom development tasks easier ; And based on OceanBase Provide development of custom components 、 Mission 、plugins Plug in tutorial , Minimize the burden of user learning and operation .

We will also optimize , Make the parameters of component configuration simpler . At present, most of the calculation component parameters of the console are Taier Custom parameters , Later, we will add parameter description to the parameters of the console calculation component , The direction of optimization is that users can know the configuration process in real time when configuring computing components .

Easy to expand

We also plan to reduce the difficulty of developing custom task types . New addition Hadoop Other computing components , Such as OceanBase、TiDB Expand the task type of distributed database, etc .

Specifically, the specific arrangements in these five aspects ：

1. Implement the type definition of custom components

2. Implement the template parameter definition of custom components

3. Develop custom plug-ins

4. Develop custom tasks

5. Customized task operation and maintenance center information display

Expand k8s Support

At present, the resource scheduling component of the console only supports by default Yarn, Cannot be used quickly Chunjun On Kubernetes , In the future, our resource scheduling component will be extended k8s Plug in support for , Reduce the difficulty of getting started in the environment of task operation .

That's all Taier The basic introduction of the console and some future planning directions , You are also welcome to exchange and discuss , Several stacks hope to grow with you .