[case] building a universal data lake for Fuguo fund based on star ring technology data cloud platform TDC

Xinghuan technology, as an enterprise level big data basic software developer , By giving full play to the hard power of basic software, it has successfully helped finance 、 The government 、 energy 、 traffic 、 Customers in manufacturing and other industries have realized digital transformation , Accumulated rich successful experience .

【 Industry case 】 It aims to regularly share the latest case achievements of Star Ring Technology , Provide reference and decision-making basis for digital transformation of enterprises in all walks of life .

This article will introduce how starring technology is based on data cloud platform TDC Build a universal data lake for Wells Fargo fund , Help it realize data unification, sharing and exchange .

Case background

Wells Fargo Fund Management Co., Ltd. was established in 1999 year , It's an old Chinese 10 One of the fund companies .2003 year , Canada's oldest bank —— Bank of Montreal, Canada （BMO） Participation in Wells Fargo Fund , Fuguo Fund Management Co., Ltd. has become the first foreign-funded fund management company among the first ten fund companies established in China . So far , The asset scale of the whole product of Wells Fargo fund exceeds  12,000  Billion , This scale is still growing . Fund companies are also data companies in a sense . Collect data 、 Processing data 、 Application data , Finally making business decisions is a typical working process in the operation of fund companies . In the process , The ability of data to transform information into value determines the competitiveness of a company , It depends on data collection 、 Handle 、 Application efficiency 、 accuracy 、 And the service mode of information . When the excess return of the industry shrinks , The information advantage determines the robustness of the return of fund products 、 Risk control ability and customer service quality . With the development of the industry , Diversification of investment varieties , The rise in asset size , Fund products have become an important tool for institutions and individuals to invest , From external supervision and other aspects related to data quality 、 Data security 、 The requirements of comprehensive data management level are also increasing ; meanwhile , Wells Fargo fund as the head company of the industry 、 Systemic financial unit , Especially in data governance 、 Data security 、 Establish a better guarantee and control mechanism in terms of data quality . However , Wells Fargo fund's original single data warehouse platform architecture began to face many challenges ：

• Real time data processing is not supported ：
  Real time risk control large screen 、 Real time direct selling redemption Application Statistics 、 Portrait and precision marketing 、 Real time valuation and other business needs are becoming increasingly urgent , Combined with the Wells Fargo Fund IT Overall planning requirements , Wells Fargo funds should build real-time data processing capabilities in due course .
• 
  

• Data warehouse platform storage is expensive ：
  Solve the storage problem of massive data , According to the data value 、 The temperature of data is different , Storage on different platform media .
• 
  

• Semi structured is not supported 、 Unstructured data processing ：
  Various research reports of fund companies 、 Notice 、 Public opinion and other external information data are also crucial to the company's decision-making , Many of these data are semi-structured 、 Unstructured storage format .
• 
  

• Data island problem ：
  Different teams build their own , The data didn't get through ; The same data is saved in multiple business systems ; There is no uniform standard , It is difficult to control data and improve quality .
• 
  

• Cloud native deployment is not supported ：
  Existing data warehouse platform  Vertica  Build on Distributed Database , Based on physical server deployment , Not flexible enough .

Solution

In view of the challenges and current situation of Wells Fargo Fund , Based on star ring technology data cloud platform  Transwarp Data Cloud  It provides one-stop data Lake solution for Wells Fargo Fund .

Based on star ring technology data cloud platform  Transwarp Data Cloud（ hereinafter referred to as TDC）、 Star ring big data development tool  Transwarp Data Studio （ hereinafter referred to as TDS）、 Star ring intelligent analysis tool  Sophon  And other products , It can solve the whole process from building the lake to managing the lake in one stop , Provide data base and analysis capability guarantee for Wells Fargo fund management and decision-making , Enhance the competitiveness of Wells Fargo funds .

Data cloud platform as the base of Wells Fargo fund data Lake cloud  TDC

 , Based on cloud native technology , With the ultimate flexibility , And integrate the data  PaaS、 analysis PaaS、 application  PaaS, Implementation data 、 application 、 Model interconnection , Break the data island of different business systems of Wells Fargo fund .

In data governance 、 Metadata 、 Data quality 、 Data resource directory and other capabilities , Star ring technology provides a one-stop tool platform integrating development and management  TDS

, Provides data development 、 Data integration 、 Data control 、 Capabilities such as data mall and data service .TDS  Our sub products cover ： Database development tools Waterdrop、 Database online development and collaboration tools  SQLBook、 Big data integration tools Transporter、 Task scheduling software  Workflow、 Visual report analysis software  Pilot 、 Data service development and management tools  Midgard And data asset catalog software  Catalog、 Data governance tools  Governor、 Data security protection tools Transwarp Defensor、 Data Mall  Transwarp Foresight、 Data service development and management tools  Midgard  And label management platform StarViewer. It can help Wells Fargo funds form standard data asset management norms 、 Build a unified data processing system 、 Sharing capability , Form data assets .

Sophon  It is a series of intelligent analysis tool software including data analysis and machine learning modeling

, Be able to integrate slave data 、 model building 、 Knowledge reasoning 、 Aid decision making , Data analysis supporting business 、 Explore 、 Service process , Thus, it can help developers implement intelligence into enterprise applications , Help enterprises carry out data-driven development 、 Operation and industrial upgrading .Sophon It mainly covers the computational intelligence in data analysis 、 Perceptual intelligence 、 Cognitive intelligence and other three main directions .Sophon The technical feature of is the use of distributed computing mode , Can be structured 、 Access of semi-structured and unstructured data 、 Handle 、 Analysis and model release and management . On the Wells Fargo fund project , It can be used to build a joint innovation laboratory , By introducing artificial intelligence 、 Data mining technology to enhance the ability of data value .

Construction effectiveness

adopt  TDC  Multi tenant capability

Realize the unification, sharing and exchange of Wells Fargo fund data

TDC Through the establishment of Wells Fargo fund unified data Lake tenant space , Centralize the data of various business systems within Wells Fargo fund , Achieve efficient governance and control of Wells Fargo fund data .

TDC  Data lake can easily realize data in different applications （ Tenant ） Shared between , Subsequent new tenants can directly use the data shared by the unified data lake ,  Tenants share some nodes in the cluster , These nodes can be all based on TDC Created tenant schedule .

TDC  Tenants can be created on demand , Resource isolation between tenants 、 Data isolation 、 Business isolation , Different tenants deploy different applications on demand . meanwhile ,TDC Support setting mutual trust between different tenants , Share components and data with each other . Computing resources among tenants can be dynamically shared , Tenant 2 Idle , Tenant 1 The computing resources of all servers of the client can be used , Provide computing performance and query efficiency . Tenants can share their own components , stay TDC Share data with others in the cloud . The sharing of tenant components can reduce the cost of component deployment . Share through tenants , Auxiliary permission control , Realize data sharing among tenants . be based on  TDC  Enterprise level data to help Wells Fargo fund build  PaaS  The platform can realize the following functions ：

*  Unify the data of Wells Fargo Fund PaaS, Big data services are all based on TDC  Empower

*  data  PaaS  The platform supports open source components 、 Extension of middleware

*  Realize multi cloud integration in multi tenant scenario

*  Improved resource utilization , Create tenants on demand , Tenants give resources to each other

*  Business isolation , Different tenants deploy big data product instances on demand

*  Unified deployment of application and data

*  Applications and data are gradually interconnected 、 The fusion

*  Build data Lake based on private cloud platform 、Hyberbase Disaster recovery scene

*  Unified data Lake space , Manage enterprise data efficiently

*  Through shared components, supplemented by permission control , Realize the on-demand sharing and exchange of data

*  Unified data application and distribution through multi tenant data lake

The end result

• Finished building dataset 、 spit 、 save 、 Count into one   Of TDC The first phase of data Lake construction , The implementation includes offline batch processing 、 Data analysis 、 Basic capabilities such as real-time stream processing .
• 
  

• Construction based on TDC The data lake of , Realize multi-source data collection , Have rich data acquisition ability , Realize the management and sharing of data among different tenants , Solve the problem of historical data storage and data island .

The ability being explored

• Continuous optimization and improvement based on  TDC  Enterprise level data to help Wells Fargo fund build  PaaS  platform , Give full play to the performance and scene advantages of big data platform , At the same time, it is supplemented by data asset management and data verification , Help improve the overall data quality of Wells Fargo fund , Realize data capitalization and enable business applications .
• 
  

• Build a data asset management platform for the whole life cycle, and finally realize the unified storage of data 、 Unified asset management 、 Unified data services 、 Unified security management and operation and maintenance , While improving the ability of data asset management , Meet the diversified needs of various application systems .
• 
  

• Quantitative investment and research , Product capability based on star ring time series database and deep understanding of distributed computing for many years , Help Wells Fargo fund improve its intelligent quantitative investment and research capability .

The future planning

• On the basis of improving data management ability , Help Wells Fargo fund build a data science platform , Realize anti money laundering 、 Public opinion monitoring 、 Artificial intelligence scenarios such as intelligent base selection .