Learn how Alibaba manages the data indicator system

The students who do the data must be right One Data I've heard about it all . but One Data The system specifically includes the contents 、 What kind of application , I wonder if you know ？ Today we share in detail One Data Contents related to data governance in the system .

—、One Data General overview

First , Let's see. One Data The overall situation of .

（1） About positioning

One Data It is the rule for Ali system to define data indicators 、 Workflow and metadata management system （ About metadata, you can   Refer to historical articles ）. It mainly includes three parts ：

• Data index specification definition ： That is to say, the data index system is regulated , This part is the core content we share today .

• Data model design and development ： In fact, it is data warehouse data model design （ can   Refer to historical articles ）, Let's not start today , Later, it will be elaborated in detail in the article of data warehouse practice

• Data research and development ： That is, the associated information of physical tables attached to all indicators returns + Algorithm service interface interfaces with external systems , I won't go into details today

From the following Ali's overall architecture diagram , Can clearly see One Data The positioning and function of the system .

（2） About the goal

One Data What are the problems we are committed to solving ？ It mainly expects to build an index and an algorithm , From the algorithm definition 、 From data research and development to data services , Traceability can be managed to avoid repeated construction .

• Index specification definition ： An indicator has only one English field 、 A Chinese field 、 An algorithm definition , Avoid the problem of different indicator logics in different departments .

• R & D process ： The reference fields of data R & D physical table structure creation and release all rely on the results defined in the indicator specification

• Data services ： A logical table with the same primary key under a topic , Only the only data index defined by the specification is provided

By providing standard （standard）、 Service oriented （service）、 Safe （safe）、 Shared （shared） data , To eliminate the pain of business and technology .

Two 、 Data index system governance

We mentioned above One Data The main contents include data index system 、 Data modeling, etc . below , Let's take a look at the overall overview of indicator system governance .

（1） Index system and examples

The overall structure of the indicator system is as follows ：

ad locum , Ali will provide a final data indicator （ Derived indicators ） Layer by layer , For example, disassemble the modifier 、 Dimensions and so on , Normalize the indicators in this way 、 systematic . Otherwise , A bunch of indicators together , It can only be an index dictionary , Not an indicator system .

The derived indicator consists of an atomic indicator 、 Time period modifier 、 Some other modifiers are combined to get .

• Atomic index 、 Modifier types and modifiers , Directly under the business process , The modifier inherits the data field of the modifier type .

• Multiple modifiers can be selected for derived indicators , The relationship between modifiers is ‘ or ’ perhaps ‘ And ’ The relationship between , Specific is determined by specific derived indicator semantics .

• The derived index only belongs to an atomic index , Inherit the data field of the atomic indicator 、 Independent of the data field of the modifier .

Atomic indicators have definite English field names 、 Data type and algorithm description ; Derived indicators should inherit the English name of atomic indicators 、 Data type and algorithm requirements .

（2） Index nomenclature

Try to use English abbreviations for terms used in names , Followed by English , When the English name of the indicator is too long , Consider naming with the first letter of Chinese Pinyin . Such as made in China , use zgzc. stay OneData In the tools , Maintain common terminology , To name .

English name of derived indicator ： English name of atomic index + Time period modifier （=3 position , for example ,_1d）+ Serial number  （=4 position , for example ,_001）. Chinese name of derived indicator ： Time period modifier +[ Other modifiers ]+ Atomic index .

Let's take an indicator 「 lately 1 God SEO Source of overseas search UV」 give an example , Look at the naming conventions of Chinese and English index names .

Only time period has English name of modifier , And the length is 2 position , add “_” Three digits , for example _1d. The list of time period modifiers commonly used by Ali is as follows :

Atomic indicators must be attached to a business process . Chinese and English name ： action + Measure .

stay OneData In the tools , Both English and Chinese names will be given by OneData Automatic tool generation .

Derived indicator to control the English name is too long , In the understanding and standardization of English names , The meaning of all modifiers is included in the serial number . Serial number is based on atomic index + Derived indicators are self increasing .

（3） Explain the related terms

Related terms about the index system , Refer to the following table for explanation ：

3、 ... and 、 Specification details of the index system
 

The following is a detailed description of the specification details of the index system . Derived indicators can be divided into three categories ： Transactional indicators 、 Stock index and compound index . According to its characteristics , Some have to create new atomic indicators , Some can add modifiers to other types of atomic indicators to form derived indicators .

Transactional indicators It refers to the indicators for measuring business activities . for example , Number of new products issued , Number of goods reissued , Number of new registered members , Order payment amount , Such indicators need to maintain atomic indicators and modifiers , On this basis, create derived indicators .

Stock index Refers to the entity object ( Such as commodities 、 members ), Statistics of some states . for example , Total commodity , Total number of registered members , Such indicators maintain atomic indicators and modifiers , On this basis, create derived indicators , The corresponding time period is usually “ History ends at a certain time ”.

Composite index It is compounded on the basis of transaction indicators and stock indicators , for example , Browse UV- Number of buyers under orders conversion rate , Some need to create new atomic indicators , Some can be based on transactional or stock type atomic indicators 、 Add modifiers to get derived indicators .

（1） Ratio type

The ratio type needs to create an atomic indicator . for example ,CTR, Browse UV- Number of buyers under orders conversion rate , Satisfaction rate, etc . give an example ：“ lately 1 Day store home page CTR”.

• The atomic index is “CTR”,

• The time period is “ lately 1 God ”,

• The decoration type is “ Page type ”

• The modifier is “ Shop homepage

（2） Proportional

For proportional type, you need to create an atomic indicator . for example , percentage 、 Proportion . give an example :“ lately 1 Proportion of daily wireless payment amount ”

• The atomic index is “ Proportion of payment amount ”,

• The decoration type is “ Terminal type ”,

• The modifier is “ wireless .

（3） Variable type

The variable type does not create an atomic indicator , Add modifiers , On this basis, create derived indicators . give an example ：“ lately 1 Day order payment amount 1 Daily variation ”,

• The atomic index is “ Order payment amount ”,

• The time period is “ lately 1 God ”,

• The decoration type is “ Statistical methods ”,

• The modifier is “ On 1 Daily variation ”.

（4） Rate of change type

The rate of change type creates an atomic indicator . give an example :“ lately 7 Days, the amount paid by overseas buyers 7 Day rate of change ”, 

• The atomic index is ” Rate of change in payment amount ”,

• The decoration type is “ Buyer territory ”,

• The modifier is “ Overseas buyers ”.

（5） Statistical

Statistical （ mean value 、 Quantiles, etc. ） Do not create atomic indicators , Add modifiers , On this basis, create derived indicators ; In the decoration type “ Statistical methods ” Add modifier under ： Per capita 、 average per day 、 Industry average 、 The average price of goods 、90 quantile 、70 Quantiles, etc. . give an example ： Natural monthly and daily average UV,

• The atomic index is UV,

• The modifier is “ Statistical methods ”,

• The modifier is “ average per day ”.

（6） Ranked

Create atomic indicators , It's usually top_xxx_xxx, Sometimes you choose rank and top_xxx_xxx Use a combination of .

When creating a derived indicator, select the corresponding modifier as follows ：

• Statistical methods （ for example ： Descending , Ascending ）;

• Ranking （ for example ：TOP10）;

• Ranking range （ for example ： industry 、 Province 、 Primary source, etc ）;

• According to what sort ( for example : Number of searches , Browse PV)

（7） Object collection type

Create atomic indicators , It's usually xxx strand ; When creating a derived indicator, select the corresponding modifier as follows ：

• Statistical methods （ for example ： Descending , Ascending ）;

• Ranking （ for example ：TOP10）;

• Ranking range （ for example ： industry , Area )

   

Examples are as follows ：

Detailed specifications for the indicator system , Let's introduce these first .

Four 、 Product design and operation

Last , Let's take a detailed look at Alibaba's product design for indicator system governance .

（1） The overall structure

The overall menu structure is as follows ：

（2） Permissions system

Have to say , Alibaba's control over data permissions is very strict and rigorous .

The process of authority allocation is as follows ：

The scope of authority is as follows ：

Role permissions are as follows ：

（3） Specification definition

lookup ： Search for + Classified Search + Filter items + Sorting mode selection + Status filter again

General situation , All of them are created through the new button in the upper right corner of the page . Modifiers , Submit audit , The system automatically passes .

Dimension attributes , Batch import is supported ：

The dimension can be managed by the attached sub dimension , Such as ： There are categories of goods , The category is the parent dimension , Product is a sub dimension ; You can view the parent-child dimension information of a dimension .

About One Data Contents related to index system governance in , Let's share these . The article also mentioned ,One Data In fact, it doesn't just include data indicator governance , There is also data modeling , We will share this related practice in detail later .