Thoughts on the construction of data analysis platform for small and medium-sized enterprises (I)

This topic is planned to be written in three parts , It includes the following ：

outline

reflection （ One ）

0） Why to write
1） General construction requirements for enterprise data platform

reflection （ Two ）

2） Obstacles encountered in actual construction

reflection （ 3、 ... and ）

3） Why not the data center
4） If you can't avoid mistakes , How to remedy

---

---

Thoughts on the construction of data analysis platform for small and medium-sized enterprises （ One ）

Why to write

   As the volume of data continues to grow , Rational use of data is an inevitable means for enterprises to improve work efficiency , About big data , Too many people have done a very detailed analysis of the data storm .
   In implementing data applications , There is an industry order . In the financial industry , Data is the lifeblood , So give priority to data analysis . The author has done data processing in the small loan industry , I have also done data analysis of industrial companies . You can find , In the financial industry , The investment in data analysis is relatively willing , A regular stroke of the pen is a multimillion dollar budget . For industrial companies , For example, those who sell electronic scales , Elevator seller , Floor vendors , We are more cautious about data input . It can be seen that , For the value of data , Or trying to cross the river by feeling the stones .
For this state , First of all, let me show my attitude ： Data analysis is valuable for every enterprise , And the investment in building the data analysis platform , Don't touch the stone to cross the river .
   Some business owners will say , My clients are all specific clients , For example, professors in schools , Graduate student , We do business in a specific relationship , No data analysis is required .
   For this understanding , I don't agree with ： Through the node time analysis of the production process , You can see the delivery efficiency of the workshop , Through the after-sales rate , We can judge the delivery quality of this batch of products , wait . I quite agree with , The economic source of the enterprise is the business department , As long as we have a good relationship with specific customers , You can maintain your income . And in customer relations , It also needs data support , For example, through the advertising cost put on the market and the region （ Or this group of people ） Compare the sales amount of , We can analyze the effect of this advertisement . So as to decide whether to participate in the same advertisement next time , Like exhibitions . Shenzhen Electronic Information Industry Exhibition , Should we go ？ Qingdao battery research and Analysis Seminar , Should we participate in the exhibition ？ In the initial stage of the enterprise , No doubt this is a must , But with some internal data of the enterprise , Is it possible to determine behavior through data analysis ？
If the enterprises listed above , All need data analysis , that , What other enterprises do not need data analysis ？
Now that it is determined that resources need to be invested to build a data analysis platform , that , There is only the question of how much to invest . Some leaders are good at bargaining ,IT The person in charge puts forward the need 200w, He said , Can it be invested in installments , The first phase will be invested first 30w, Look at the effect . This behavior is actually suitable for 80 In the s, all industries were waiting to prosper . Now look at ,30w It also depends on the effect , That's it “ Few people , Time is short , Fast production ” The needs of . There is a couplet ：
   Upper couplet ： This demand is very simple
   Lower bound ： I don't care how to achieve it
   Streamer ： Go online tomorrow
   about IT Ministry , It is too simple to produce an analysis report , Nothing more than extracting data , Data cleaning , The report shows several links .
   I don't care how to realize it , It's simpler , Make an interface that supports manual entry , Import historical report data , forbid ？ Then enter the result manually , The manual entry shall prevail .
   however , about IT For the Department , It takes a week to produce a report , And produce 1000 Reports , if necessary 1000 Zhou , That would be too much .
   therefore , What we need is a data analysis platform that can quickly produce reports , Since it is to build a platform , You will be 200w Bargain to 30w, It's just killing programmers . Programmers to catch up , Regardless of platform construction , Lead to no standard , No specification , Poor efficiency , Repeat the wheel , Even regardless of safety , Regardless of performance , Such a platform , Like a castle in the air , crumbling . Individuals will buy accident insurance , Enterprises are not even willing to invest in data security , That would be too much .

General construction requirements for enterprise data platform

   Many enterprises do not know what resources are needed for the data analysis platform , The following is a brief introduction to the general construction requirements of the data analysis platform , Not the standard , It's just casual talk
   There are three main processes ：
   1） Understand business needs
   2） Building a data warehouse
   3） Report output

1） Understand business needs
   This is the basis for building a data warehouse , If the business is not analyzed in detail , The built data warehouse will not meet the needs of various reports , Finally, sew and mend . For different companies , The methods of business analysis vary , You can communicate with the line owner , Head of Department , Obtained from the interview with the person in charge of the direct business , Then the business language is transformed into design language to promote to programmers .
2） Building a data warehouse
   Data warehouse It is based on the understanding of the business , structure Subject oriented , Integrated , Relatively stable , A collection of data reflecting historical changes . According to the usage of the business , Set up a data set for specific user groups to provide data analysis and rapid report output , That is to say The data mart . During construction , Usually divided into ODS/DWD/DWS/ADS Four levels of construction , To meet the data cleaning 、 transformation 、 Summary 、 polymerization 、 Dimensionalization 、 Personality analysis, etc .
   Building a data warehouse （ Several positions ） In the process of , Several aspects need to be considered ：
  2.1） The data is accurate ： The data is not accurate , It will bring difficulties to the follow-up analysis . In the business system , such as ERP, Listener , Financial cloud and financial system , There is inaccurate data entry , In manual reports , There are data records without constraints . These data need to be stored in the data warehousing stage , Just clean it , Ensure the quality of warehouse data . I remember when I was making statements in a small loan company , because 6 The data difference of gross money , Quarreled with the team leader for hours , Finally, the data can be corrected by re cleaning . If the data is not accurate , The report made , No matter how beautiful , Nobody's using it . Besides, , In the initial stage of making and promoting automated reports , In fact, it increases the workload of the corresponding department , that , Accurate data is crucial for promotion . Because in the parallel stage of manual report and automatic report , They can proofread data through system reports , This reflects the significance of the report system at the initial stage .
  2.2） Uniformity ： You need to build a unified dimension table and fact table , Guarantee consistency . During data cleaning , You also need to ensure logical consistency , For example, some account conversion logic , The logic of the data cleaning phase , Both need to be unified and confirmed by the business .
  2.3） Rapid output capacity ： When making reports , It needs the ability of data warehouse to provide rapid response , Including the simplicity of data retrieval logic and the speed of query , All are quick response . Both meet the needs of rapid development , It should also meet the requirements of front-end rapid response . therefore , It is required that in the data warehouse stage , Just slice the data 、 clustering 、 Dimensionalization , Increase redundancy and other work .
3） Report output
   In making reports , Yes Finereport,BIEE,EasyReport You can choose , At present, the more applicable estimate for small and medium-sized enterprises is Finereport 了 , be relative to BIEE It's cheaper , Compared with open source reporting software , More standardized , It can be used immediately . When making reports , See too many people write unmaintainable code , Poor performance , In this way, when there are few data and users , Can barely meet the requirements , And the number of users has increased , Or data increase , That will require rewriting the code , This is unscientific . Data warehouse should also provide fast and convenient data , To meet the needs of rapid report output and user-defined analysis .

---

---

Last , Talk about data security .
   Some time ago , Helped a company do Oracle Data recovery of database , I was shocked by the construction standard of its database . He built one by himself 32 nucleus /128G Memory servers , Let me install a single machine Oracle 11G, No backup , No, RAC Redundancy such as clustering , The data table is named randomly , Let all the restored data tables be Chinese headers . It can be seen that , No matter from the data platform , To program development , To data security , Did not pay enough attention to . And this data recovery , It is because the previous server failed . The reconstruction , Or continue to run naked . This data analysis platform , It is better to analyze the report manually .
   The so-called weak foundation , The earth trembled and the mountains swayed . In the use phase , If the server goes down , Or the disk is damaged , Or low level of programmers , Deleted the library and ran away , Will be given to the entire department , Even a great blow to the whole company . When relying on data , There's something wrong with the database , That's the big problem .Oracle Because considering downtime , A lot of work has been done on the data reading and writing logic , And in use , Users actually ignore these possibilities .
   Bad disk track is a common problem , Programmer misoperation is also common , So in redundant backup , Permission control , Prevention and control should be done well in advance , The damage caused by an accident is immeasurable .