How to correctly understand the real-time nature of Bi?

We usually talk about business intelligence BI The mechanism of data synchronization is generally T+1 Pattern . Just like the stock market T+1, Stocks bought that day , Sell on the next trading day, such as tomorrow . So this T, Some people understand it as Today today , perhaps Trade Trading day , No problem .

If it's like a weekend 、 If you don't trade on holidays , It means Trade Just a trading day . For business intelligence BI for , This T It can be understood as Today same day ,T+1 Data is synchronized once a day , The data we see now is from the first day , Look at today's data tomorrow .

therefore , business intelligence BI There is a lag in data synchronization , Not real-time , However, this method can completely meet the business intelligence of most enterprises BI appeal .

business intelligence BI The real-time demand of

Speaking of this, I think of a trading company we have been to before , The general agent of a product in China , The business is growing well . We are talking about business intelligence BI Of T+1 Data synchronization mechanism , The other person's CIO And the boss are chasing this problem . The page data must be completely real-time , Second level real time , Try to look like tmall 、 double 11 That big screen , Ask if you can do .

Data visualization screen  - Parker data business intelligence BI Visual analysis platform

This question is not a very positive answer , It is very difficult to be completely real-time , No one can fully guarantee . Because this is not a business system , This is business intelligence BI Analysis system , The analysis system has the time loss in the data retrieval process and the time loss in the index calculation process . So our consultants didn't respond positively , Still explaining business intelligence BI How is it realized .

The other person may be impatient , Just cut in and say you don't want to explain these things , Just answer if we can do it , We are going to implement this function . Your business intelligence BI If you can't do it , Just say it directly , Don't cover up . If we can achieve , Money is no object .

The feeling of listening at the scene is actually not very good , And especially if money is not a problem, it is always a problem of money in the end . Out of respect for customers , We still promise to go back and discuss the technical implementation scheme before making an offer . We later learned that this company has found several other big data companies 、 business intelligence BI company , Including a large factory , That's what I asked .

Because the customer has no clear budget , Then we can report as we should , As a result, the customer said that it was too expensive to do anything after the report . Let's fill in 、 Several large screens 、 A dozen statements you made are so expensive . Yeah , Just a little bit , You have to do something real-time 、 Engage in big data architecture , Also evaluate hardware and software resources , This is a typical blind toss .

Data visualization screen  - Parker data business intelligence BI Visual analysis platform

Of this company CIO It has also become the bottleneck of the information construction of the whole company , Because it is obvious that many basic technical principles have not been understood , business intelligence BI I still didn't understand the explanation . about CEO Persistent wrong ideas have not been corrected in time , Inadequate assessment of needs and inputs , This kind of communication is a serious waste of time and energy for both parties .

Why? business intelligence BI use T+1 Pattern

Let me talk about why business intelligence BI use T+1 Pattern , What is the data processing process like .

business intelligence BI The data warehouse architecture itself determines that the requirements for real-time data are not so high ,ETL The process of ,Extraction extract 、Transformation transformation 、Loading load , These three links themselves are time consuming .

business intelligence BI - Parker data business intelligence BI Visual analysis platform

First ,Extraction extract , These business data need to use SQL perhaps ETL The tool queries from the business database and loads it to business intelligence through network transmission BI Data warehouse , Depending on the size of the data, this query and loading will take some time , Like five minutes 、 Ten minutes or more . This is just one of the queries , All queries and loads are executed in a few minutes , It's normal for a long time to be an hour .

second , Data loading into business intelligence BI Data warehouse , Data must be processed , Such as going to a heavy 、 Merge 、 Cycle calculation, etc. calculate various indicators and put them into different layers of the data warehouse , This data processing process is in business intelligence BI Is the most time-consuming , Dozens of minutes to hours . More indicators , The more complex the business logic is , The longer the calculation process takes .  This is ETL The core of processing ,Transformation transformation .

Third ,Transformation The converted data should be written to the target table, such as business intelligence BI Middle dimension table 、 In the fact table , namely Loading load .

Data warehouse  - Parker data business intelligence BI Visual analysis platform

ETL It's business intelligence BI The key of data processing in data warehouse , among E Is the data source ,T It is the process of intermediate data calculation and processing ,L Is to write the calculation results to 、 Load to target table .

Let's see. , Add up the time of these three stages , Is this time period very long , There is no way to achieve business intelligence BI For real-time analysis .

In addition, some people may use the business system late in the daytime , Some data business departments update later . So most of the time , business intelligence BI Will choose in the evening 12 Execute after ETL From the business data source 12 Click to synchronize all the updated data before .

therefore , Usually business intelligence BI In the evening, I dealt with the first two stages , All the data is extracted 、 machining 、 The calculation is finished , Are stored in the data warehouse in advance . The next day, when the page is refreshed, it directly accesses the calculated data of the data warehouse , So it's fast .

Data visualization  - Parker data business intelligence BI Visual analysis platform

If you come across another person who asks about business intelligence BI Real time data , You can tell the other party about this process , Under normal circumstances, you should still be able to understand the process .

There is no way to achieve indicators in business intelligence BI Real time analysis on visual pages 、 Real time presentation , Or quasi real time , Welcome to update , In the next article, let's talk about .