Description of the functional scenario of "collective storage and general governance" in the data center

科技飞速发展的时代,企业信息化建设会越来越完善,越来越体系化,当今数据时代背景下更加强调、重视数据的价值,以数据说话,通过数据为企业提升渠道转化率、改善企业产品、实现精准运营,为企业打造自助模式的数据分析成果,以数据驱动决策.

After data from China to unify the data,会形成标准数据,再进行存储,形成大数据资产层,进而为客户提供高效服务.支持海量数据的存储、计算、产品化包装过程,For the front desk based on data customization of innovation and for the continuous evolution of the middle business based on the data feedback to provide strong support.China can solve data island、Loss of data assets、Insufficient data service ability、The problem of low data values,最终Can enable data to assign the business scenario、产生业务价值. 

1整体介绍 

数据中台分了三个模块：数据分析平台（DAP）、主数据治理平台（MDM）、数据集成平台（ESB）,Data middle deployment environment usingK8S云平台部署,MDM实现基础数据管理,业务数据由源头业务系统提供,通过ESB的数据集成,实现基础数据、业务数据从源头系统到数仓的同步. 

1.1产品方案 

数据中台是基于公司的三款核心产品DAP、MDM和ESB组成的解决方案,主要实现基于数据治理分析的中台建设,通过MDM将企业内部的基础数据标准化、规范化、整合化后,由DAPFor warehouse construction and final data analysis, visualization show,ESB通过数据集成完成整个数据中台的主数据分发和数仓的数据汇聚. 

1.2集成架构 

介绍一下DAPThe data of China program integration framework： 

数据中台方案方案主要的产品组合是DAP数据分析平台通过与ESB应用集成平台结合使用场景主要是通过配置调度任务,通过执行调度任务调用ESB数据同步流程,实现数据的采集、抽取、转换、传输、调度等操作,基础数据为DAP提供同源并标准的、一致的数据保障数据分析分析数据的准确性,然后通过Portal门户集成平台展现给客户. 

1.3方案介绍 

Data middle main实现各个业务系统数据的有效整合,数据中台把数据统一之后,会形成标准数据,再进行存储,Form data assets layer,进而为客户提供高效服务,Support data storage、计算、产品化包装过程,For the front desk based on data customization of innovation and for the continuous evolution of the middle business based on the data feedback to provide strong support,To improve the enterprise internal data and the connection between the application and coordination,降低成本并提高使用速度,实现企业业务集成和数据集成,Make integration more agile、Let customers to use more convenient.This article from the data of China“集”、“存”、“通”、“用”、“治”USES the Angle of the deep analysis of China. 

2整体介绍 

通过数据中台的“集”、“存”、“通”、“用”、“治”打通企业的数据环节,实现全生命周期的数据管理,通过数据中台建设数据管理体系,实现各个业务系统数据的有效整合,通过基础数据治理保证底层基础数据的一致性,基于业务指标进行前端的动态展现,结合数据指标的多维度穿透,Implement different form、不同维度的分析展现. 

2.1数据采集 

Data collection mainlyAccording to the need to collect relevant data analysis corresponds to the theme of the,Provide the basis for data analysis,Data analysis of data from other system data within the company,According to the needs of different data sourceESBThe enterprise service bus (esb) or direct callDAPThe methods of the interface,Availability of the analytical data、全面性. 

 

2.2数据存储 

Data storage is oneThrough the data from the various business systems to build data warehouse,In terms of data analysis and decision support provide service for the user or machine learning,After collecting and processing data generated set persisted to a computer,So as to enhance the value of data,Help enterprises to make decisions analysis. 

 

2.3数据互通 

Information system integration are gradually achieve a more extensive connection,万物互联Will be the future development trend of the integrate.Among all the systems within the enterprise to integrate,The data management,实现数据的同步和分发,实现信息共享,At the same time to realize the integration of the enterprise internal,Through business boundaries,To realize ecological.

2.4数据应用 

最终The valuable data in the form of visual display,Can effectively help enterprises to clearly analyze advantages and disadvantages,从而调整企业策略,加快企业的信息化发展与整体竞争力,And can be released to strengthen the data through the data service availability,Guarantee the accuracy of the data application party can get full data. 

  

2.5数据治理 

如何Help data accurate、完整,This requires the data data governance,Through the middle management enterprise data,梳理业务流程,Can break the data island,Provide strategic support for the development of,Lay the data base for the subsequent information. 

3数据采集 

The above said data acquisition isIntegration of various business system data,The business system data source more and more、越来越复杂,In addition to the traditional application system,The rise of the Internet has formed more extensive external channel,In the face of such complex data source,How to select the data systematically、Collection is the next to explain. 

3.1功能说明 

Data acquisition is data collection step,The business system of the data collected in the middle,Data into business data and basic data,Data is the basisMDMBasic data management platform for data acquisition of the synchronous to the platform for data management,The business data is throughDAPSelect business systems of different data table,采集到ODSAmong libraries are defined and selection,Acquisition process are throughESBApplication integration platform to realize heterogeneous now、Distributed system connectivity between. 

3.2 功能价值

Data in China in the process of collecting data throughESBPreset various adapter component,Connect each existing isolated application system,以图形化、Drag-and-drop build integration process and service,Improve the efficiency of application system integration work,Reduce the risk of a heterogeneous system integration.In view of the future possible business requirements change,On application integration platform for dynamic adjustment,Implement the application system integration strategy between smooth upgrade. 

3.3功能展示 

1.Basic data acquisition process 

The source of the first data modeling configuration of master data system： 

Each data precast synchronous interface： 

First of all to invoke synchronous temporary table interface,Synchronous to the temporary table before： 

数据导入功能： 

Click on the button synchronization to the real table or a synchronous call real table interface： 

接口API说明： 

Primary data collection to data management in the： 

2.Business data acquisition process： 

Data acquisition process from the business system toODS的过程,First of all define application system： 

And then define the database： 

进行ODS定义：ODSSource mode is divided into source library to read（一般是MDMBasic data platform of master data table）、流程同步、接口推送. 

1.源库读取：Make the business system from the library,Will directly through the business system synchronization to warehouse. 

2.接口同步：Business system call interface synchronization is the interface for data synchronization. 

3.流程同步：Process synchronization is withESBWith the generation scheduling process of data synchronization. 

ODSProcess synchronization is divided into a timestamp and field contrast. 

Below is the field contrast：Field configuration： 

Can also be configured by timestamp query：Need to configure the timestamp field. 

The following is to choose the corresponding information,Through integrated interface to generate the corresponding process. 

点击flash后,Generates scheduling resources： 

By scheduling tasks clicking perform,Will call this process,Then the data synchronization toODS中：

集成结果： 

4数据存储 

After the data collection to better for enterprise use,就必须Data stored in the data in the middle,And thus for the data application,Only strengthen the focus,They can better turn data into assets. 

4.1功能说明 

Data storage is divided into basic data and business data storage,Basic data storage is the synchronized master data synchronous interface toMDMManage product,The business data stored byODS采集后,For construction of the warehouse,通过ODSTransition to the number of warehouse steps to establish a dimension table、事实表、汇总事实表,In creating a number of silo models. 

4.2功能价值 

Data and data between natural there are explicit and implicit relationship,Data of acme attractionThrough these relationships recognition and mining,Create an unprecedented application scenario,Bring unexpected huge value.And in order to realize all this,First of all need to data gathered on the physical plane,Let the valuable information automatically、Quickly integrated into the unified storage space,Data for the subsequent development、Data analysis, lay a solid foundation for. 

4.3功能展示 

1.基础数据存储 

主数据管理： 

参考数据管理： 

类别列表： 

编码列表： 

业务数据存储： 

通过ESB把ODSData stored in the number of positions in the dimension table、基础事实表、汇总事实表： 

Below is the dimension table configuration,The source of the dimension table table isODSTable so click select table will enter the correspondingODSTable select page,选中相应的ODS表后,You can write back to front page,如下： 

After completed the corresponding data after,Here in order to limit the dimension table must select source table add corresponding limit,When select source list and then click the save,In order to more humanized pop-up hint whether directly as we have selected tables add corresponding relationship,如下： 

选择ok进入点击页面,We can for the two tables below fields set corresponding relationship,之后,Click ok input corresponding relationship.And home state is adjusted for editing style,如下： 

We can be found in the edit state details page we configure the relationship.如下： 

Because we chose the source table in detail page,So the field information page in field function,Only can choose in your source of strain field in the table,具体样式如下： 

Table name drop-down selected here only we choose in the above table name,如下： 

After selected will appear the corresponding fields in the table,Again after entering the choice to choose the field will not choose again,At the same time delete function is found in the field information page,When we deleted by importing the field function of import,This field will be recovered in the original form,如下： 

Below is a basic fact table configuration process the fact table can undertake choosing according to different types of the corresponding source table,First select basic fact table,Basic things table source can be a dimension table、ODS,如下： 

Configure the mapping field and source： 

存储效果： 

5数据互通 

Among all the systems within the enterprise to integrate,The data management,实现数据的同步和分发,实现信息共享.At the same time enterprises inside and outside、Integration of online and offline,To be able to get through the business boundaries,To realize ecological.

5.1功能说明 

Through the master data management enterprise internal unified data standard,Give priority to data standardization provides a unified platform for the centralized management,For all the information interaction unified coding data are provided for the integration and data exchange.In the process of master data management,Enterprise service bus (esb) as an auxiliary tool,Complete the data synchronization with the distribution of. 

5.2功能价值 

随着互联网的发展,Required by the business connected not only limited to traditional forms of business integration between internal integration.The business enterprise is the linkage between unified exist at the level of demand,Only starting from the most basic data,打好底层基础,After the system function of the information integration between,Block boundaries,Finally realize the linkage of the overall business process,才能To avoid the generation of information island,Achieve the interconnectivity of enterprise internal information. 

5.3功能展示 

Basic data data distribution： 

Basic data distribution home page by using distributed configuration of master data and field： 

Configuration distribution field： 

Through configuration distribution permissions,Configuration of master data packet can distribute the service system. 

Call distributionAPIInterface to invoke a unified distribution： 

通过BPMFlow through each business system： 

6数据应用 

Data from China construction is a comprehensive system engineering,The nature of the data in China isSupport fast hatch data applications,Storage capacity was calculated by the technical connection data,用业务连接数据应用场景能力的平台. 

6.1功能说明 

By analyzing multidimensional technology realize multidimensional、多角度、A full range of real-time online analysis,Including data procession transpose、Drill linkage、Layers of penetration and other analysis style,To help managers to grasp company operation situation overall,The data show section provides diversified、Enrichment of presentation,包括但不限于饼图、柱状图、折线图、气泡图、面积图、省份地图、词云、瀑布图、Funnel figure such as cool chart. 

Data used another service,数据中台集成业务系统数据,一般情况展现层的数据不是本身系统的数据,是不可能再去业务系统抽取,那样的抽取的性能会变差,因为业务系统数据已经到数据中台中而且是治理后标准数据,所以应用From the data in the middle,而获取的途径就是DAP数据服务,Data visualization and other products is also can get effective data through data services show. 

6.2功能价值 

通过数据可视化,Business organizations canTo improve their ability to search the information you need when needed,并且比其他公司更高效地完成这些工作. 

6.3功能展示 

1.数据可视化应用： 

Through the different components show different data,In the different side、Different resolutions on show different analysis chart： 

PC看板： 

大屏展示： 

Mobile display： 

  

2.数据服务应用: 

Business data provides data services： 

查询服务： 

统计服务： 

指标服务： 

业务服务： 

7数据治理 

Data in order to highlight the value of China,To carry out treatment to the data of China,构建标准化、流程化、规范化、一体化的数据治理体系,Effective data management can ensure that enterprises fully consistent credible,To fully enhance the value of middle data assets. 

7.1功能说明 

Data governance can be divided into basic data management and business data management,Basic data governance is throughMDMBasic data management platform for data cleaning、管理、For examination and approval of operations such as help achieve basic data consistency、完整性、准确性. 

Business data management mainly fromODS到数仓（维度表、基础事实）Process quality inspection,Help the integrity of business data,So when a summary calculation can help the data accuracy,To set up several warehouse model. 

7.2 功能价值

Data for the enterprise's important assets,With the advancement of enterprise information,The data between the enterprise system cannot be effectively exchange and sharing,影响数据的实时性、一致性和准确性,形成了数据孤岛.对于企业来说,Valuable data of each independent system is not only the data,But the system integration between the sharing of data,The emergence of the data island hindered the integration and sharing of the data.Through the middle management enterprise data,梳理业务流程,可以打破数据孤岛,Provide strategic support for the development of,Data foundation for the follow-up information. 

7.3功能展示 

基础数据治理： 

数据清洗：Through the data cleansing function based data import business system toMDMCheck the cleaning,After cleaning in the imported into the master data in the table real.  

Data checking function：Existing data through data checking function repeated data selection. 

业务数据治理： 

配置校验规则：Configure the regular expression validation rules. 

Dimension table field configuration associated validation rules：  

Scheduling process check： 

质量日志： 

8心得总结 

无论是现在的互联网企业,还是传统型企业,都需要数据分析,数据分析的目的是把隐藏在一大批看来杂乱无章的数据中的信息集中和提炼出来,从而找出所研究对象的内在规律. 

8.1平台结合 

有效的解决方案一定是According to their own business case for the combination of product split打造出来的,Data usually middle management scheme andESBEnterprise service bus (esb) products in combination with,MDMResponsible for data governance,ESBBe responsible for the data synchronization distribution.The choice of several flow associated to any project,Will abide by the plan reasonable planning、The optimal combination of product model,In the enterprise information ability or business requirements have not achieved the standards on the basis of building,The reasonable use of existing resources,Avoid to cause unnecessary waste of resources. 

8.2 平台价值

数据中台集成业务系统数据,一般情况展现层的数据不是本身系统的数据,是不可能再去业务系统抽取,那样的抽取的性能会变差,因为业务系统数据已经到数据中台中而且是治理后标准数据,所以应From the data in the middle,而获取的途径就是DAP数据服务. 

And another kind of visual data applications show isThe data a way of application,通过数据可视化,Business organizations canTo improve their ability to search the information you need when needed,并且比其他公司更高效地完成这些工作通过数据可视化,Business organizations canTo improve their ability to search the information you need when needed. 

8.3平台发展 

With the deepening of the data process of middle construction,Analysis to the traditional data analysis and data processing technology has brought about many challenges.Cloud computing and open source technologies to promote the development of the data to the ground,分布式存储、The relational database and parallel processing technologyGradually become the key technology in the process of big data project implementation. 

All data collected were analyzed,Analysis with data from sampling data extended to all;其次,Analysis using data from the data in the field of traditional single extended to cross domain data,After the big data can be different in the field of data combination analysis;再次,Related data based on the analysis of the data source to predict the correct results.