Where is the big data open source project, one-stop fully automated full life cycle operation and maintenance steward Chengying (background)?

Link to the original text ： Three minutes into kangaroo cloud one-stop fully automated full life cycle operation and maintenance manager ChengYing（ Undertaker ）

Courseware acquisition ： Official account “ Several stack Study Club ”, Backstage private message ** “ChengYing”** Get live courseware

Video playback ： Click here

ChengYing Open source project address ：github 丨 gitee Like our project, give us a ** STAR！STAR！！STAR！！！**（ Important things are to be repeated for 3 times ）

Technical exchange nail qun：30537511

2022 year 5 month 30 Japan , Kangaroo cloud has opened a one-stop fully automated operation and maintenance manager ChengYing（ Undertaker ）, In order to help community students better use ChengYing, Since the open source, we have been working hard to prepare for the live broadcast , from 6 Month begins , We will organize a live broadcast for you once a month , Share with you ChengYing Related content of , You are welcome to continue to pay attention .

One 、ChengYing brief introduction

ChengYing（ Undertaker ）—— It inherits the concept of the famous sword family of the kangaroo cloud open source project , Taken from the shadow sword of the top ten famous swords .

ChengYing（ Undertaker ） As a one-stop fully automated full life cycle operation and maintenance manager , Provide one-stop deployment of big data products 、 Operation and maintenance 、 Monitoring service , It enables product deployment 、 Product upgrades 、 Version rollback 、 Expansion and contraction nodes 、 Log diagnosis 、 Cluster monitoring 、 Real time alarm and other functions , Committed to maximizing O & M cost savings , Reduce the online failure rate and the difficulty of operation and maintenance , Provide customers with safe and stable product deployment and monitoring .

ChengYing Born out of kangaroo cloud data stack, it is a one-stop operation and maintenance manager EasyManager, From open unified monitoring 、 To define standardized deployment capabilities , Then introduce Prometheus/Grafana/ Since the research dt-alert Components , Complete unified monitoring 2.0 Function optimization of , And then to the newly increased cluster management capabilities , Finally, the internal incubation was completed , Successful open source ChengYing.

Two 、ChengYing features

Introducing ChengYing Before the functional features of , Let's familiarize ourselves with a few concepts ：

Noun concept

• product

Referring to ChengYing Deployed big data products , Such stack （DTinsight）、 Cloud log （EasyLog） etc. , As ChengYing The highest level organizational unit .

• Components

In general, and " Component installation package " equivalent , It refers to that the product contains multiple components , That is, the product contains multiple component installation packages , Such stack （DTinsight） The product includes Hadoop、DTBase、DTCommon、DTBatch、DTStream Wait for multiple components , Represents different applications .

• Service group

Service classification under component , Such as Hadoop Components contain HDFS、Spark、Yarn、Flink、Default Wait for the service group , Differentiate Services .

• service

Refers to the specific services under the service group , Such as HDFS The service group contains hdfs_datanode、hdfs_journalnode、hdfs_namenode、hdfs_zkfc And so on ;Spark The service group contains hivemetastore、spark_historyserver、thriftserver And so on .

• Host grouping

It refers to the artificial grouping of hosts , When the number of hosts is too large, it can be grouped , Easy to manage .

• host

The server , Including physical machines 、 virtual machine , Refers to the hardware resources required for product deployment , Usually with the host IP Or host name .

features

ChengYing It has six functional features ：

• A unified Schema Definition

Abstract package definition , Use a standard Schema Define the complete lifecycle of a package , Including installation 、 start-up 、 To configure 、 upgrade 、 Unloading, etc .

• Cluster management

Support multi cluster settings , Convenient in one set ChengYing Deploy multiple sets of resource isolation services in the system ; Support for all product packages in the cluster 、 All hosts 、 Cluster resource management .

• Installation and deployment

Support the rapid installation and deployment of product packages .

• Service management

Support to view the operation of various services and service instances 、 Health check status ;

Support service configuration modification and distribution ;

Support rolling restart of services ;

Support service configuration view .

• Monitoring alarm

Integrate prometheus/grafana Components monitor clusters as visual dashboard components 、 host 、 Service performance test ; Support email 、 SMS 、 Nail and user-defined alarm channel send alarm information .

• Node expansion / Shrinkage capacity

Support according to the cluster health , The production supports the expansion and reduction of host resources .

3、 ... and 、ChengYing System architecture design

• ChengYing System architecture

• ChengYing Agent Design

• Unified Schema Design

• Monitoring design

• Explanation of architecture design

As the content of architecture design is more , Here we will not make a specific review , You can watch the video to learn more .

B Station live review address ：

https://www.bilibili.com/video/BV1Ht4y187xo?spm_id_from=333.999.0.0

Four 、ChengYing Quick start

Next, I will introduce how to deploy quickly ChengYing：

• Environmental requirements

• Rapid deployment

Download installation package ：

https://github.com/DTStack/chengying/releases/download/v1.0.0/chengying-release-1.0.tar.gz

Unzip the installation package ：

tar zxvf chengying-release-1.0.tar.gz

cd chengying-release-1.0

Run the installation script ：

sh install.sh local_ip(local_ip by Chengying Location machine ip)

view pages ：

http://local_ip

• Upload component installation package

• Select cluster and installation package

• Configure the service

• Execute deployment

5、 ... and 、ChengYing The future planning

Finally, let's introduce ChengYing Future planning and main things to do in the near future ：

The future planning

• Service Orchestration

Support automatic orchestration based on host role and service type , Reduce labor costs .

• Deployment way

Support customized deployment package pipeline sequence .

• Safety certification

Support cluster startup Kerberos authentication , Bill generation and download .

• Monitoring alarm

Support based on PromQL Custom alarm settings for , Rich alarm types .

Key issues in the near future

• plan 8 Release at the end of month ChengYing V1.1.0 Release.

• Live broadcast of open source technology once a month .

• And Chunjun/Taier Dream linkage -- Try to use ChengYing Deploy Chunjun/Taier.

Kangaroo cloud open source framework nail technology exchange group （30537511）, Students interested in big data open source projects are welcome to join us to exchange the latest technical information , Open source project library address ：https://github.com/DTStack