Production practice | Flink + live broadcast (1) | requirements and architecture

Want to make APP Same thing as WeChat , Can run small programs smoothly ？ | Experience will send you to Xinjiang 、 Huawei 、 Cherry keyboard ！>>>

productive practice | be based on Flink Live real-time data construction of （ One ）| Requirements and architecture

*

Each article in this series is based on some practical production requirements , Solve some problems in production practice , throw away a brick in order to get a gem , To help the partners solve some practical production problems . I believe everyone has watched the live broadcast more or less , Have you ever thought about , If you are responsible for the construction of the company's overall live real-time data , How to build it ？ This series of articles mainly introduces the whole process of live data construction , If it helps your partner , Welcome to thumb up + Look again ~

*

First think about a few questions

• 「WHAT： I believe everyone has watched the live broadcast more or less , Even if you are an anchor or responsible for the business is related to live broadcast , Have you ever thought about , In live business scenarios , What indicators do you care about most and what you need to focus on 、 What data to build ？」

• 「WHY： Why do we need to build live real-time data ？ Can't offline construction be satisfied ？」

• 「HOW： How live real-time data enables business ？ How to divide the live broadcast real-time data according to the needs of the company's live broadcast scene ？ How to build live real-time data body ？」

• 「WHO： In the process of building live real-time data , What kind of components are needed to build ？ Each component is responsible for which part ？」

Let's start with the above questions ~

live broadcast + Short video , The next battleground for content operations

With the development of Internet technology , More and more people pay attention to webcast , Live broadcast after a gushing explosion a few years ago , Recently, the heat has decreased . The homogenization of content and the difficulty of realization are the main problems faced by live broadcasting , With the popularization of mobile terminals and the acceleration of network , Short video quickly obtains each big platform by the short and fast big flow transmission way 、 Fans and capital , So the live video starts with a lot of features . meanwhile , Some are mainly developed by short video app Also added live broadcast function in the software , Live and short video make up for each other , Exist side by side and play a part together , It brings users a better use experience , It also brings more traffic to the major platforms ," live broadcast + Short video " Has become a new development trend .

This series of articles focuses on the construction of live real-time data . This is the first article in this series , Requirements and architecture , It is divided into three parts , In order 「WHY - WHAT - HOW」, From these three angles , Answer the first three questions , among 「WHO」 Part of this series of articles in the follow-up construction details of the chapter are introduced ！

WHY： Why build live real-time data ？

Compared with the production and consumption of short video , The link between the host of the live broadcast and the audience watching the live broadcast is established in the live broadcasting room , The interaction between each other is only produced in the studio , And usually , The duration of a live broadcast is just a few hours , Therefore, the production and consumption timeliness of live broadcast is stronger than that of short video , Therefore, the demand of live data for real-time is higher .

WHAT： Need to pay attention to 、 What to build live real-time data ？

Need to pay attention to 、 What to build live real-time data ？ In other words, it is based on 「 Data analysis business needs 」 set out , Decide what kind of live real-time data to build ？

Live broadcasting is a link between the host and the audience , All the operations are carried out around the host and audience , Students of data analysis will carry out analysis from this most basic point of view , So first of all, we can follow the data of the whole live broadcast according to 「 Live production 」 and 「 Live consumption 」 Into the most basic division .

In addition to this angle , Data analysis students will also learn from 「 Global live broadcast business insight 」 and 「 Single studio insight 」 Analysis insight on different particle sizes , So it's also possible to follow 「 Big data 」、「 Single studio data 」 division .

From these two angles , It can basically cover the demand for live broadcast business analysis scenarios , Therefore, live real-time data can also be divided and constructed from these two perspectives .

To sum up, the whole 「 Live real-time data service division and enabling application architecture 」 As shown in the figure below .

 

Business division and Application Architecture

among

「 Live broadcast real-time data 」 On the macro level, monitor the live broadcast business , Provide the ability to forecast the market ; Among them, the minute granularity time series can quickly locate the peak time of each behavior in the live broadcast , Detailed attribution can be based on that moment . besides , When the live broadcast is doing operational activities , It can also quickly see the effect of operational activities based on real-time data , Real time optimization of enabling activity strategy .

「 Live broadcast of real-time data in a single studio 」 You can monitor the live broadcasting service of a single live broadcasting room in a fine-grained way , It is used to output live data war reports during live broadcast 、 In addition, it can evaluate the real-time effect of single live broadcast room and reasonable allocation based on the effect of data war report .

Detailed live data requirements and examples are shown below .

The broader market

「 Production side 」

• 「 indicators 」： The total number of live broadcasting rooms ...

• 「 dimension 」： Live studio portrait 、 Host user portrait

• 「 give an example 」：[ The live broadcasting room is for live broadcasting of games ] Of [ Total number of anchors on the air ]

「 Consumer side 」

• 「 indicators 」： The overall audience watched 、 give the thumbs-up 、 comments ...

• 「 dimension 」： Audience user portraits 、 Other dimensions of log reporting

• 「 give an example 」：[ At present, I watch live broadcast in Hebei Province ] Of [ Total audience ]

Single studio

「 Production side 」 A single studio is usually filled with portrait information , So there are fewer such indicators , Let's not discuss it for the moment .

「 Consumer side 」

• 「 indicators 」： Single studio audience watch 、 give the thumbs-up 、 comments ...

• 「 dimension 」： Audience user portraits 、 Other dimensions of log reporting

• 「 give an example 」： Some studio [18-23 Age group ] Of [ Total audience ]

At present, we have learned what the real-time data of live broadcast contains , Now it's time for a big fight .

HOW： How to build ？

How to build ？ In other words, from a technical point of view , How to 「 Live broadcast real-time data business needs 」 Turn into 「 Technical scheme of live real time data 」 To land ？

From a technical point of view , The above requirements for live real-time data construction can be summed up in one word ：「 Live real-time multidimensional indicators 」.

Multidimensional

That is, the output index is multidimensional , Including public dimension and non-public dimension .

The first category is 「 The public dimension 」. It consists of three parts , Live studio portrait , Host user portrait , Audience user portraits , The word "public" means that such dimensions can be shared by multiple indicators . give an example ： After the broadcast in a studio , The studio portrait only needs to be built once , It can be repeatedly used by multiple indicators , Not only can be used as a large side production 、 The dimensions of consumption indicators , It can also be produced as a single studio 、 The dimensions of consumption indicators .

The second type is 「 Non public dimensions 」. Non public dimension is bound with specific consumption behavior , That is to say, it is bound to a certain indicator , Dimensions reported with log escalation . give an example ： The client type of a viewer watching the live broadcast （ Android ？IOS？）, When watching the live broadcast, the province and other dimensions , This kind of dimension is only related to the current consumption behavior , Can't be shared by other metrics .

 

Multidimensional

indicators

All of them are pv,uv Class index . Simple understanding is corresponding to each dimension xx The amount .

 

indicators

Real time data construction technology architecture

Corresponding to the live real-time data construction process mainly includes two parts ： The public part and the non-public part .

The public part is the construction of real-time public dimension table .

The non-public part is the index non-public dimension and corresponding production 、 Consumption index construction .

Give the total directly 「 Technology Architecture 」 chart , The following articles in this series will introduce the detailed reasons for the overall architecture design .

 

Technology Architecture

Simple description .

The data source includes the production side , Consumer side data sources ;

The data processing part includes the construction of public real-time dimension table , And index building , Some of the public dimension tables are also supported offline ;

Finally, the data collection part , The production side , Multidimensional indicators on the consumer side are used by data analysts .

Notice of next section

The next section mainly introduces 「 Live broadcast of the construction of real-time public portraits 」, Among them are... In the technical architecture diagram 「 Anchor users 、 Focus on user portraits 、 And the studio portrait 」 The construction plan of .

summary

This paper first puts forward several problems about the construction of live real-time data . Trigger with these questions , There are three sections .

The first section briefly introduces the reasons for the strong timeliness of live broadcast , Therefore, the demand of live broadcast for real-time data is even stronger .

The second section starts from the angle of data analysis , It leads to the content of live real-time data that we need to build , And from the market / Single studio , production / The consumption angle has carried on the module division .

In the third section, the overall architecture design of the technical scheme is carried out for the data requirements .

The last section summarizes the paper .

If you have the same building requirements or you have built Live Live real-time data , Welcome to leave a message or leave a link to your article , Communicate with each other ~

**

official account (mangodata) Reply from Li flink Keywords can be obtained from flink Learning materials and videos .

**