Application practice | Shuhai supply chain construction of data center based on Apache Doris

Reading guide ： Shuhai supply chain is a set of sales 、 Research and development 、 purchase 、 production 、 Quality assurance 、 Storage 、 transport 、 Information 、 Finance as one of the catering supply chain service enterprises , Because its business is relatively complex ,2020  By the end of the year  Apache Doris  Upgrade the core architecture , And in  2021  Started construction in  Apache Doris  As the core data center . This article will be from data access , Data service arrangement , Data security ,Doris  Application and so on .

author

｜

Head of Shuhai supply chain big data team   Wang Yongxu

Business background

Shuhai supply chain is a set of sales 、 Research and development 、 purchase 、 production 、 Quality assurance 、 Storage 、 transport 、 Information 、 Finance as one of the catering supply chain service enterprises , To provide catering chain enterprises and retail customers with overall food supply chain solution services . Because its business is relatively complex ,

2020  By the end of the year  Apache Doris  Upgrade the core architecture , And in  2021  Started construction in  Apache Doris  As the core data center .

In the use of  Doris  Before , We used  CDH  This data platform , Many components are used , But the link is too long , And the development and maintenance costs are relatively large , Finally, there is no good  OLAP  System .

Because our data history burden is relatively light , Yes  Apache Doris  Research and testing , decision

Use to  Apache Doris  Build a data platform for the core , It has the following advantages ：

• At the same time, it supports high concurrent point query and high throughput  Ad-hoc  Inquire about .

• At the same time, it supports offline bulk import and real-time data import .

• Both details and aggregate queries are supported .

• compatible  MySQL  Agreements and standards  SQL.

• Support  Rollup Table  and  Rollup Table  Intelligent query routing .

• Support better multi table  Join  Strategy and flexible expression query .

• Support  Schema  Online change .

• Support  Range  and  Hash  Secondary division .

• High availability , It can tolerate some nodes hanging .

• Simple operation and maintenance , Deploy , maintain , Upgrading is relatively simple , Independent of external components .

The architecture is as follows ：

Because of the previous understanding of

Metadata , Data services , Access data quality , The construction of kinship

Introduced , This article will be from data access , Data service arrangement , Data security ,Doris  Application and so on .

Data access

Data access function is an important part of data development , We have developed a data access system , stay  Web  End operation ,

Realize zero code data access to  Doris, The following is an introduction to the main functions ：

• subscribe  MySQL Binlog, Warehousing to  Doris  surface .

• subscribe  Kafka Topic, Warehousing to  Doris  surface .

• Data dynamic cleaning , Writing code on the page can complete the conversion before data warehousing .

• Access task merging , To save resources , Support sub database and sub table access in one task , Support multiple  TOPIC  Access in a task .

• Dynamic data quality verification , Configure field quality rules , Check the access data quality .

• Warehousing encryption , During re connection , You can encrypt sensitive data and then enter  Doris  surface .

• Error data management , Because of network or data errors and other reasons , Data re warehousing can be completed on the page .

• Data access link monitoring , For example, error data monitoring , Abnormal monitoring of data production link , Abnormal monitoring of data consumption link , Task data access trend chart , Trend chart of cluster data access .

Data access task list ：

Data access task configuration ：

Data access dynamic code processing ：

Data service arrangement

Data services are used by business systems  API  A system for obtaining data . It can be done on the page  API  newly build 、 edit 、 Online development debugging 、 Set current limiting 、 Online and offline operations . because  API  There may be a business logic relationship between , And you cannot configure the same  API, We developed the data service orchestration function , By dragging and dropping , Give Way  API  Can arrange and transfer data between , External provision  API  when , What is still exposed is a  API.

give an example ：

The user and the city to which the user belongs are stored in one  MySQL  data source , The sales volume of each city is kept in  Doris  data source . To be developed  API  The function of is that users can only view the sales of their cities . Then it can be realized through the service orchestration function ,Node1  Nodes pass through users  ID  Get the city ,Node2  The node gets the output of the upstream node ( City ) As input , Get city sales as  API  Output .

The input and output of each node can be customized , Input can come from  API  Request parameters , It can also come from the output of an upstream node , It can come from global parameters , Such as user  ID, Paging parameters ;

Data security construction

Data security is a big topic , It's all about ,

Here from data encryption , Data authority and data warehouse data backup are briefly introduced .

Data warehousing encryption

In the process of data access , You can choose to encrypt fields , When connected to  Doris  After the table , It is already encrypted data , Follow up data analysis , You can use the key to decrypt .

Data access encryption configuration ：

Data access

Because the company has a wide range of people who view reports , For the same data model , Sales in every city and every region , operating , Factory personnel , The data viewed by managers and other personnel is different , You need to accurately control row permissions and column permissions , therefore ** We are  Doris  The upper layer developed a set of Data permission system , Through configuration , Complete data permission configuration , It can be accurate to row permissions and column permissions .**BI  The reporting system acts as an access party , Introduce the data permission client and implement the corresponding abstract method .

give an example 1：

For a report model , Zhang San can only view data in North China or northwest China ; Li Si , Wang Ming can only view Xi'an or Beijing , And the sales volume is greater than  10000  The data of ; Zhang Si , Zhang Wu is not restricted , Others have no authority .

List of model row level permission rules ：

Row level empowerment rule editing ：

give an example 2:

  Everyone can view the report data , But everyone can only view their own city , And the amount is greater than 200 Or the amount is less than 100 The data of .

Free combination rule conditions and rule relationships ：

Personnel label management ：

give an example 3：

Column permission rules , You can set forbidden viewing for users , Data desensitization and other rules

Column level permission configuration ：

Data warehouse backup

We use  Dori  As the core of storage and Computing ,Doris  The data itself has been stored in multiple copies , But considering disaster recovery , We will still back up the core access data to  HDFS, Therefore, a data warehouse data backup system is developed , hold  Doris  The table is divided according to the total quantity or partition , Scheduled backup to  HDFS.

Backup schedule configuration ：

Backup scheduled task list

Doris  Application

We use it  Doris  It carries the calculation and storage of data analysis . Besides , There is also a scenario like this ： Business  MySQL  Database data has been growing , A large amount of historical data affects online performance , And it cannot be deleted directly , Because there are also low-frequency historical data queries , So ,

We are based on  Doris  Developed a set of business historical data archiving system , The historical data that will not be changed can be archived incrementally at regular intervals

, Provide data query through data service system , Push archived data to the business side , The business party verifies , And delete historical data .

Archive plan list ：

Archive plan configuration ：

Data push plan configuration ：

earnings

At present  Doris  As the core data platform , It has supported the data query and data analysis requirements of dozens of business systems of the company . by BI Intelligent analysis , Each business system provides excellent query performance , And greatly reduce the data platform maintenance , Data development , The cost of data center construction .

• The real-time data access is stable and reliable , adopt  Stream Load, Thousands of watches are accessed in real time , The total number of data accessed every day is at the level of 100 million , Very stable and reliable ;

• Support high concurrency and high performance data online analysis and query , Every day to  Doris  The number of online analysis queries is at the level of millions , Most of the  SQL  At the millisecond level , slow  SQL  There is also a lot of room for optimization , also  Doris  It will automatically optimize queries in some scenarios ;

• By directly querying the original access table , Establish a materialization attempt , Index , It supports multiple real-time query requirements with low latency and high concurrency . And many tables  Join  Excellent performance ;

other ：

• Doris  The overall structure of is simple , The operation and maintenance cost is very low , Online rolling upgrade , It can save manpower and focus on the construction of data center and business development ;

• Doris  Highly compatible  MySQL  agreement , Interactive query analysis , Provide an efficient data development experience ;

• High availability , Data partition multi copy storage , The overall service will not be unavailable due to the exceptions of some nodes ;

• Wide ecological compatibility , The community provides and  Flink,Datax  Wait for big data interaction  Doris  plug-in unit , adopt  Broker  Importing and exporting data is simple and fast ;

• The community is active ,Doris  Functions and performance are constantly expanding and improving , If you encounter problems, you can get close help from the community .

Join the community

Welcome more partners who love open source to join  Apache Doris  Community , Participate in community building , Except in the  GitHub  Ascending  PR  or  Issue  outside , You are also welcome to actively participate in the daily construction of the community , such as ：

Join the community

Solicitation activities

, Perform technical analysis 、 Applied practice and other articles ; Participate as an instructor  Doris  Community online and offline activities ; actively participate in  Doris  Questions and answers from community users

SelectDB  Is an open source technology company , Committed to  Apache Doris  The community provides a full-time engineer 、 A team of product managers and support engineers , Prosper the open source community ecology , Create an international industry standard in the field of real-time analytical databases . be based on  Apache Doris  R & D of a new generation of cloud native real-time data warehouse  SelectDB, Running on multiple clouds , Provide users and customers with out of the box capability .

Related links ：

SelectDB  Official website ：

https://selectdb.com

 (We Are Coming Soon)

Apache Doris  Official website ：

http://doris.apache.org

Apache Doris Github：

https://github.com/apache/doris

Apache Doris  Developer mail group ：

[email protected]