Notes on standardized management of "ancestral warehouse" of meituan meal

Construction history path

A period of rapid growth

2010 To 2016 year , Hundred regiments war

Business ： The main business strategy is mainly actual combat , Single business , The analysis scenario is fixed

Core challenges ： Fast response to demand

Construction ideas ： Vertical construction

Data governance ： Basically not , Restart after failure

mature period

Business ： The business idea is to increase revenue , There are more realizable products , The scale of products undertaken has increased 2 To 3 times , The workload expands rapidly

Core challenges ： Ensure quality while efficiently supporting demand

Construction ideas ： Standardization construction ideas , Unified architecture , Avoid repeated construction , Improve efficiency

problem ： The efficiency of standardization construction is very low , Strict requirements are not strictly implemented

Optimization idea ： Experience and standards must be standardized and productized

Development period

Business ： Refined operation , cost reduction , Innovative business

Core challenges ： With the same scale and demand , Non functional scenarios should be improved , Delivery speed 、 The query efficiency 、 Lower storage and computing costs

The core idea ： The system automatically helps solve the problem , Data warehouse construction is not limited to specific needs , Build for multi scenario analysis .

Metadata service capabilities and data service capabilities are packaged into products and provided externally , Products such as diagnostic recommendations depend on the index relationship in metadata .

problem

Poor metadata quality , The document is unreliable and even does not , What data is not clear .

Poor code quality , There are many online faults .

The cost of architecture expansion is high , Expansion was not considered in the initial design , Technology upgrade costs （ Iterate according to big data technology ） May be lower .

Test monitoring is for single job , Quality depends on R & D capability , There will be problems when code handover occurs .

Poor metadata quality

problem ： The field is repeatedly defined

solve ： Define the standard metadata system .

Code quality

problem ： The quality of model definition is poor , There are English and Chinese

Extensibility

problem ： There is no end-to-end testing , Technology upgrade （ Iterate according to big data technology ） Can get some promotion , If the upgrading of data warehouse architecture cannot adapt to technology upgrading , Will be eliminated .

There will be repeated construction of indicators in the construction of application layer , But developers themselves may not find it , Are developing their own .

solve

Metadata definition

Index classification , Standard syntax defines metrics ： Basic indicators 、 Derivatives 、 Calculate the index

Set up a data governance committee , Sort out the overall business process , Form a bus matrix .

The Chinese description is consistent , The modifier rules are consistent .

Sort out historical models and field names , Then do word segmentation , Analyze high-frequency words , To define specifications , Form a field Library .

Organize English words 、 The frequency of the occurrence of roots , High frequency can be used as a standard , The review passed , It can be translated automatically according to Chinese .

Quality management

A large number of calculation rules have been accumulated in the process of continuous demand in history , You can sort out all possible combinations of calculation rules and strategies , Commonly used in iteration DAG Sort out the picture .

Put the calculation rules sql Convert to configuration , Let the business personnel configure , Count yourself .

Technology Architecture

Computing engine and application engine , Produce slow upgrade computing engine , Query slow upgrade application engine , Two lines are self closed loop , Do not affect each other .

Construction of logical model ： Construction of market level , The application related physical table is changed to logical table , The upgrade logic model of the application layer does not change . The logical model of the application layer is managed uniformly , The system can cross compare and verify .

End to end monitoring

If you find an exception, mark it on the page first , Mark the abnormal conditions and reasons .

planning

Fuzzy search of metadata , Intelligent search , Intelligent guidance .