Version 2.0 of tapdata, the open source live data platform, has been released

One minute quick understanding  Tapdata

6 month 29 Japan ,

Tapdata

  The product release and open source briefing will be opened online , around 「Your Last ETL」 This theme , Closely 「 real-time data 」 This word eye , Official announcement comes with  ETL  Real time data platform  Tapdata Live Data Platform  go online , as well as  Tapdata  Heavy news such as the open source plan of core functions .

Press conference site ,Tapdata  Core team members and many data industry experts 、 Pioneer of data ecology 、 Representatives of open source data products 、 Representatives of enterprise clients and investment institutions gathered , Focus on “ The flow of ”、“ caller ” data , Combine historical solutions and practical cases , Stand on the production 、 Different perspectives such as consumption and capital , Discuss real-time data application scenarios and technological changes , In depth analysis of the technical architecture of the new generation of real-time data platform , Have an in-depth insight into the development status and cutting-edge advantages of the data industry , Bring continuous dry content and intensive sharing of wonderful ideas , Energy is constant , Now let's take you back to the highlights of this activity .

△  Click to watch the full video playback

One 、 Why does the era need a new real-time data architecture ？

The data demand of offline analysis scenario has happened in the past , The data requirements of real-time business scenarios are clear in the future . And the scene difference is enough to breed a new technical architecture .

Growing data islands and limitations of historical solutions

In the past decades , Enterprises have built many business systems , And the number is still expanding . With the increasingly low coding and platformization of data architecture , Enterprises can create more new business systems at a lower cost . As a result, more and more enterprise data and systems , This directly leads to the problem of data islands . Because the systems are not connected to each other , The process of accessing data becomes complicated , Before actually using data , Much more needs to be done “ additional ” The job of , Including data access 、 Integrate 、 Integration, etc .

Common patterns of enterprise data integration include traditional  API、ETL, In the early  ESB, And today's mainstream  Kafka, Under the influence of many existing schemes , A large number of various data integration links have been generated within the enterprise . These schemes meet some technical requirements while , It also inevitably exposes limitations in the evolution of the data age .

among ,

API  Integrate

The cost is relatively low , As long as you have certain code ability , No need for third party tools , That is, the R & D team can perform the system according to the data sharing requirements  API  encapsulation , Provide data for new downstream businesses . But build directly on the source library  API  It also has a great impact on performance , And  API  Usually there will be  Rate Limit, It is difficult to support massive data reading and writing . Besides ,API  Basically, you can only publish data for a single database , Difficult to operate across libraries .

ETL

  It is also the main promotion scheme in the past for a long time , The advantage of this approach is , Don't write too much  Java  Code or service code , But through tools or scripts , To realize the extraction and replication of data to downstream systems .ETL  The limitation of is mainly reflected in the difficulty of management , Because it is too simple and cannot be reused , As a result, every new business needs a lot of  ETL  link , Eventually scattered throughout the enterprise .

The end of lack of unified management , It's the painful spaghetti structure . Face this pain , Good architectural solutions emerged 20 years ago —— use  

ESB/MQ

  Push data to centralized enterprise message queue 、 On the service bus , Then, the enterprise systems that need to share data , adopt  API  and  Service  Connected in a way , It saves the repeated work of interaction between multiple systems , It reduces the docking cost between systems . But the overall cost is still high , So it is mostly used in commercial programs . And the development is complex 、 The system coupling is high , Low performance , The fever soon subsided “ chrysanthemums after the double ninth festival ”, Be similar to  Kafka  Such distributed open source products are replaced .

About ten years ago ,

Kafka

  Quickly caught on , A large number of enterprises began to be based on  Kafka  Realize data integration . But because of  Kafka  Not born for this , Initially, it was just a distributed log storage , Therefore, its architectural design features are more inclined to high concurrency 、 High performance 、 Distributed . Compared with the data integration requirements, the link is short 、 Short time 、 Short delay , be based on  Kafka  Of  ETL  The architecture has many nodes , Instead, long links appear 、 Data is easily interrupted 、 Troubleshooting is difficult . If you want to achieve , Much more needs to be done  Java  Code development , High complexity of use .

Last ten years , Various

Centralized data platform

Cheek by jowl , Especially with  Hadoop  The main big data platform , And traditional data warehouse 、 New algebra warehouse and other representatives , The performance of this kind of scheme is to centralize the data scattered in various data islands in the enterprise into one platform , So as to achieve unified access to the required data through the central platform . However, its technical architecture is mostly based on  Hadoop, In essence, it is also a technology stack with offline analysis as the core scenario , It is mostly used for insight into historical data 、 analysis , The data is not real enough , Unable to support some of the higher requirements for real-time data  TP  Business scenario .

After comprehensively reviewing the technical architecture and limitations behind many existing solutions ,Tapdata  Start thinking about the possibility of solving the data island problem in a better way —— Do it for the last time  ETL, It's also real-time  ETL, Realize data virtualization through data mirroring , And centrally store the mirrored data , After certain processing , Form a reusable data  Copy  Model . Then on this centralized platform , Provide the freshest demand data for downstream businesses in various service-oriented ways , In essence  DaaS  The realization of concept . Based on this ,

Tapdata  We have developed a complete set of production plan ：

Tapdata Live Data Platform

.

Tapdata  Choose to completely self-study

There are so many excellent open source components today ,Tapdata  Why not build solutions based on these excellent organizations , But designing a new architecture ？

indeed , Using the model of open source components can indeed solve some problems to a certain extent , But not the best choice .Tapdata  The reason why I choose to develop a new technical framework by myself , In addition to making the product small and light 、 Besides better maintenance , It also includes its own technological cognition and pursuit .

In a real case scenario ,Tapdata  Find out ,

Real-time business scenario （OLTP）

Demand for data and

Traditional offline analysis (OLAP)

Have essential differences . In real-time business scenarios , Data demand is generally second ; The data itself participates in the core business flow , Every piece of data is linked to the real business , High unit value , The requirements for data accuracy are  100%; The volume of business data is much smaller than the offline analysis scenario .

How to better adapt to the characteristics of real-time business scenarios , At the same time, it meets the needs of traditional offline analysis scenarios ？Tapdata  be based on  DaaS  Architecturally Live Data Platform  Or it will be the optimal solution of the current era .Tapdata Live Data Platform  The point is  Live, Lies in the flow and liveliness of data , The technical architecture of this integrated real-time data service platform , It is also designed to fit this theme .

Two 、Tapdata Live Data Platform： The first one is based on  DaaS  Architecture of real-time data platform

The fact proved that ,DaaS  It reflects a very natural and reasonable service trend , That is, data is abstracted into services , Provide easy-to-use data capabilities for all downstream businesses .Tapdata  Our vision is to build a system based on real-time data  DNA  Of  DaaS  platform , Let users not pay attention to the underlying technology , Just focus on business logic and data —— Like running water , Turn on the tap and you can use fresh data , It should be simple ！

Tapdata  Born in a time when everyone is right  DaaS（Data as a Service） There is a period of understanding but little touch on its essence . At the time , take  DaaS  There are few data products as infrastructure , but  Tapdata  Determined this direction , And continue to work in this direction . It took three years , Carefully polished the first one based on  DaaS  Architecture of real-time data platform ——

Live Data Platform

（LDP）.

In three years, the edge has first appeared ： from  DaaS  The forerunner brings his own  ETL  Real time data platform

Step away from anchoring the real-time data track  DaaS  The first step is to start , It took three years ,Tapdata  The data service products in the target have begun to take shape . So what scenarios can we use  Tapdata LDP  To complete what kind of work ？

scene 1： Real time data integration platform

Can be  Tapdata LDP  As a real-time data integration platform （Real Time Data Integration）, To replace and upgrade the old generation  ETL  Tools , Like  Kettle、OGG、Kafka ETL  The plan is relatively complicated , or  ETL  Script etc. .

scene 2： Real time data service platform

Upgraded real-time data service platform （Incremental DaaS）, It can be used to build an enterprise level data sharing center , Put the core data of the enterprise into the centralized platform , replace  ESB/MQ, In some scenarios, it can also replace  Hadoop  Big data platform 、 Data warehouse, etc , Can better serve the enterprise  BI  Data analysis and other downstream businesses provide data service support .

scene 3： Real time master data service

future ,Tapdata  Real time master data services will also be launched （Active Master Data Service）, You can upgrade the traditional master data update solution on a weekly or monthly basis ,Tapdata  Our goal is to update master data in real time , And with the help of  API  The service is directly connected to the tourism business , So that downstream businesses can directly use enterprise core data .

Tapdata LDP  How it works ？

As shown in the figure above , On the left is all the data sources of the enterprise , Including mainstream  OLTP  database , And business systems 、 file 、 Information flow events and so on . Tapdata LDP  The working mechanism of ：

First step , First, based on the ability of log parsing , Through the open framework  Plugin Framework, In real time , Modify these data sources at the first time / to update / Collect and standardize the changed data , Enter the stream processing framework after forming the standard time ;

The second step , adopt  Tapdata  Self research plan , No need to leave the process , Data calculation can be completed in the process 、 Modeling and transformation , Quick results , Get into  DaaS Storage  layer ;

The third step , Putting data into  Storage  When the layer , In fact, a set of logical models has been formed , ad locum , Users don't need to care where the data is stored , Just focus on what data information you really need ;

Step four , It's also  DaaS  The key value of , At the service level , There are two mainstream data service models , Namely  Pull  and  Push. The former refers to  Tapdata  Will automatically publish some  API, these  API  Support low code release , Data can be released according to specific needs . When the required data has been stored in the business system , You can go through  REVERSE ETL, Put these in reverse 、 The data of governance is pushed to users , This is the same.  Push  Pattern .

Through the above figure, it is not difficult to find , All data-driven businesses are on the far right , It is not included in  Tapdata  Within the four step workflow . Because no matter what kind of business you want to do with data , They are all things that users need to pay attention to ,

Tapdata  Focus only on providing accuracy 、 Consistent latest data . This is it.  DaaS  The essence of ： We don't do business , We only prepare for real-time data

.

Tapdata  Technology Architecture

As shown in the figure below ,Tapdata  The technical architecture consists of a large plug-in system 、 Data management 、 System and interaction are three parts .

among , The plug-in system also includes the integration of data and structure 、 Computation and operator integration 、 Cache integration and other components ：

• Data integration ： It is used to connect various data sources , Including real-time database integration based on real-time log parsing, which is the main function of the platform 、 With  API  and  Webhook  Real time integration of featured services or applications 、 come from  Excel/TXT  And other documents , As well as from  Kafka  And the integration of various message queues . at present ,Tapdata  In the platform
  Yes  40+  Support of different data sources
  , Include  Oracle、MySQL、PostgreSQL、SQLServer  Isomainstream database 、API、 queue 、 The Internet of things etc. , And continue to expand more data sources and types .

• Structural integration ： It is inseparable from data integration , Provide an observation view of the data structure , It can help users better understand their own data , Understand the flow and changes of the data structure itself .

• Computation and operator integration ： Corresponding engine components , To undertake the  LDP  The calculation function of . yes  Tapdata  A computing component designed for real-time data service scenarios . In the engine , Users can complete dragging 、 Low code building  ETL  The task of , It can also be based on  JS Python  And so on , Visually encapsulate various operational components , It can also realize the heavy high-order ability of multi stream wide table flow aggregation , Easier to use .

• Cache Integration ： Corresponding cache storage , yes  Tapdata  Components specially designed based on the unique demands of stream computing scenario for storage , It is a contradictory combination of order and randomness , Represents a high balance between performance and accuracy .

• Data source plug-ins ： After completing the data calculation , Data can be written into the target library required by users through the data source plug-in , Complete the circulation closed loop .

• Data API： utilize  Tapdata  Built in data storage service , It can also directly publish the calculated data as  API  Interface , Achieve the real data as a service .

stay

Data management

part , For users' demands for data exploration and perception ,LDP  Provides data traceability 、 Search and data directory functions .

stay

System and interaction

Part of , The system module focuses on enterprise features , Including monitoring alarm 、 All staff audit and other modules ; The interactive part provides browser based interface operation 、 Interactive command line operation , And immigrate  SDK  Integrated operation of , To meet the needs of different users .

A preliminary understanding  LDP  After the framework , The following will analyze the details of its design principles based on several key modules 、 Problems encountered in construction and corresponding solutions .

Tapdata  The core of the architecture is self-developed

• Plugin Framework： Design of plug-in platform extension system

Tapdata  Each component in the plug-in system corresponds to different extension capabilities ,“ Scalable ” This principle is embodied in  Tapdata  Every aspect of Architecture Design , It brings convenience to adapt to different scenes . so to speak  

Tapdata  It is a platform growing on the plug-in system

.

Select several representative data connection plug-ins （DataSource Plugin） For example ：

① DATA： Accurate, efficient and robust data access

Real time scenarios not only require high performance and accuracy of data access , It can be checked under various abnormal conditions 、 The requirement of recoverability is also high . So ,Tapdata  Besides common interfaces such as batch flow integration , It also provides many interface methods that are not common but very useful , Jointly support a real-time data platform that can provide enterprise level data accuracy and stability guarantee . for example ：

• Full incremental breakpoint continuation ： Make mistakes occasionally ,  Fast recovery

• Data playback ： Mistakes have been made ,  Fast rollback

• Get the latest event time of the source library ： Accurate delay judgment

• Unconditional heartbeat broadcast ： Avoid the breakpoint effect of sparse events

• Idempotent design :  Ensure the final accuracy of the data

② META： Elegant model auto inference system

On the issue of data accuracy , Outside the data itself , The accuracy of data structure cannot be ignored . As the number and type of data sources continue to expand , Heterogeneous replication and other scenarios are right  Tapdata LDP  The maintenance pressure will only increase , Compilation costs will also be higher and higher . For this question , There are two common solutions . One is based on  JSON  Type or language native type for structural description , Its disadvantage is that it often leads to inaccurate heterogeneous synchronization table structure , You need to manually adjust . The second is based on some open source frameworks , When using, you need to manually create a table on the target side , Before you can start doing calculations . Obviously, it's not a very good plan .

So ,Tapdata  Give a creative new idea —— Introduce an abstract middle layer , Just describe the mapping of the data source to the middle tier , The most suitable target type will be automatically matched , Give the mapping relationship , And automatically build the target table model . After the system goes online , The problem of inaccurate meter building encountered on the user side has been greatly reduced , At the same time, it also fundamentally solves a major problem of expanding the number of data sources .

Poke it here , View the finished product effect demonstration of the data connection plug-in , Show me a  MySQL  The whole process of data source from registration to use

• Incremental Engine： Distributed lightweight real-time computing engine

Tapdata LDP  The full name of our computing engine is  Incremental Engine, seeing the name of a thing one thinks of its function , Designed for incremental real-time computing , It is also the most suitable engine architecture for real-time data services that we have found so far . The engine is an integrated design , Compared with the traditional multi process data exchange mode ,

Tapdata LDP  Make the link extremely simple .

Source → engine → The goal is , All work is done in one process , It greatly reduces the burden of users . Minimalism does not mean functional compensation ,LDP  The engine has complete functions . From basic synchronization 、 transformation , To advanced multi stream merge 、 Multiple aggregation calculations ,LDP  The engine has the ability to support ; Data sources 、 Calculation 、 operator 、 Storage can also realize plug-in expansion . Besides , Between multiple computing engines , It also realizes the automatic failover of tasks , It has the advantage of distributed high availability . And in the scene of multi stream windowless merging ,Tapdata  On the premise of meeting the accuracy of data, the resource consumption is reduced dozens of times , This is also  LDP  A characteristic ability of .

The engine is  Tapdata LDP  Provides the core driving force

, It perfectly meets the needs of the computing framework of the real data service platform .

• CDC CACHE STORE： Cache engine optimized for streaming computing scenarios

CDC Cache Store  Components are designed to cache  CDC  Incremental Events . As shown in the following table , Several common databases on the market , Performance in incremental subscription , There are some defects more or less , It can't well meet the needs of real-time data service scenarios .

Facing the contradictory demands of sequential subscription and random read-write proposed by stream computing on storage engines ,Tapdata  Choose to abstract cache storage , We have developed a cache engine , On the basis of random reading and writing with certain high availability , The concurrency performance of event definitions has been greatly improved , While supporting the platform to handle more budget tasks , It also makes the technical architecture of the whole platform cleaner .

• API Service： Create an end-to-end complete closed-loop product solution

In the face of real-time data as precious as gold ,Tapdata  Strive to maximize the convenience of data acquisition . As an integrated solution ,LDP  The following capabilities have been achieved ： The wide table is directly published as an interface , Help the business go online quickly ; The upper abstraction of database type , Shielding interface differences ; Support online interface debugging 、 function ; Support enterprise level management , Audit and monitoring functions are available …… thus ,Tapdata LDP  Let the data value play a perfect closed loop , This is also  Tapdata  One of the design goals of .

Tapdata  Advantage characteristics

As a new generation of data platform ,Tapdata LDP  It has the following three characteristics ：

①  Clear service-oriented architecture ： Use data in the most efficient way

Service oriented architecture means , We can synchronize the data to the central platform for reuse , This greatly reduces the data link from the source , Reduce the number of links from hundreds to tens or even less , Reduce the impact on the source pool . From the perspective of real-time data integration ,Tapdata LDP  Need fewer nodes , It can be reduced from a dozen processes to twoorthree processes . And the boundary is clear , Focus on the first kilometer or the first few kilometers of data , Refuse the whole family bucket , Focus on basic abilities .

②  Full link real-time capability ： Support  TP+AP  scene , Greater data value

This is also  Tapdata  Flagship  DNA  The quality of , from  LDP  You can see the name of  Tapdata  Yes  Live  High regard for .Live  Namely fresh 、 fresh ,Tapdata  The whole process is for the highest value  TP  And real-time analysis  AP  scene , It aims to play a greater value of real-time data , Mainly reflected in these aspects ：

• Collect real-time ：Tapdata  Support super  40+ Data sources , Support source to target  Any to Any  Data real-time synchronous docking , Next, we will open source , stay  Tapdata  Besides the main force , Let developers participate in CO creation , Work together to quickly expand the data source to  100+.

• Transmit real time ： From the source end to the target end , Precise control , The transmission delay as low as sub second level is realized .

• Computing real time ： When calculation is required in the process ,Tapdata LDP  It has the ability to process tens of thousands of real-time streams per second , In the case of a single node , This capability can be further improved through parallel distribution .

③  The easiest data development experience ： For developers 、 For data engineers

From the perspective of examining a data product ,Tapdata  Very concerned  LDP  Ease of use and flexibility , There is no need to deploy more than a dozen nodes , developer 、 Data engineers can download it directly and use it very conveniently , Create excellent use experience .Tapdata  There are two ways to use interaction ：

• Visualization of the whole process ： For data engineers , It supports the processing of all enterprise data in a pull-up manner 、 modeling 、 Handle 、 Merge , What you see is what you get , Get a permanent real-time updated data model quickly .

• Initiate  Fluent ETL API： For developers , Especially for the open source community , There is no need to  SQL, You only need to write a program to have the ability of data integration , Complete data development .

Tapdata  Believe in ,IT  Service is a very clear trend , from  20  Years ago, Amazon started using infrastructure as a service （IaaS, Infrastructure as a Service）, More than ten years ago, database middleware was used as a service （PaaS, Platform as a Service）, In recent years, it has been particularly hot  SaaS（Software as a Service）,“ As a service ” It's developing very fast , The value of service has also been proved by history . Today's trend , Is to abstract data into a service , Provide support for all downstream businesses .

Making data use is as simple as turning on the tap and using tap water , This is a  Tapdata  Vision , It's also  Tapdata  The original intention of naming ——Make Your Data on Tap.

meanwhile , We are also pleased to find , More and more attention has been paid to this track in the industry , Expect to gather more forces to promote the development of data technology further .

Tapdata LDP  The self-service experience channel has been opened

Read here , I'm sure you're right  Tapdata LDP  There's a certain amount of awareness . Want to know more about it ？ Want to really get started  LDP  Product function ？ Welcome to register as  

Tapdata LDP  The first batch of experience officers

, Get the exclusive service of experience officer ：

Self service process ：

The first 1 Step ： Click here （ Join hyperlink ：

https://tapdata.mike-x.com/wrD1a

） Register as “Tapdata  Experience Officer ”

The first 2 Step ： Enter after successful registration “ Experience officer personal Center ”

The first 3 Step ： Click... In the exclusive service “ Download the installation package and get the enterprise version  License”, Follow the trial instructions to complete the installation and  License  Activate . Or click on “ Enterprise online  DEMO  Experience ” Get login account and password

3、 ... and 、「 fish 」 And 「 Bear's paw 」 Have it both ways ：Tapdata  Keep pace with open source and commercialization

Tapdta  founder  TJ  From write down  Tapdata  Start with the first line of code , I decided to open the source code .

6 Last day of the month ,Tapdata  The open source version was officially launched .

Whether from the perspective of the company or from the perspective of industry ecology , Open source and commercialization have never been “ fish ” And “ Bear's paw ” The relationship between , They are complementary to each other . Open source brings technological innovation , by  Tapdata  Deliver endless iterative power ; The benefits of commercialization feed back to the open source community , It will also provide a stable basic support for the continuous operation of the open source model .“ to open up ” And “ Open source ” It is  Tapdata  Carve in  DNA  Strategic adherence in .

• GitHub  Project links ：https://www.github.com/tapdata/tapdata

• Open source sites ：https://tapdata.github.io

Tapdata  Why open source ？

today , More and more enterprises and users have realized the value of data . But complex and cumbersome existing open source solutions often deter many users . Facing the existing data platform, open source solutions have long links 、 Not in real time 、 The high cost 、 Difficult to maintain and other defects ,Tapdata  We are trying our best to create a kind of fast 、 real time 、 Simple 、 Easy to use new platform .Tapdata  Hope to be based on fully automated real-time data integration capability , Connect and unify enterprise data islands , Become the master data base of the enterprise .

At the same time, we also know ： A person 、 The power of a company is limited , and  Tapdata  Our vision is ambitious , On the way to the goal , There are a lot of challenges waiting for us . We need the voice of the community 、 User's encouragement 、 Active participation and multi-party help , Work together to build stronger products . We hope that through open source , Let more and more developers participate in  Tapdata  In the use and development of , help  Tapdata  Open source projects achieve faster growth , Meet the demands of more users faster , Let more users get the value of fresh data , And bring

more

Requirements and scenarios .

Tapdata  founder TJ  Start by writing the first line of code , We decided to open our source code . We are ready to —— Give Way  Tapdata  Open source version , Provide fresh blood for more developers to deliver real-time data .

Tapdata  Open source  RoadMap

As shown in the open source roadmap above ,6  month  30  The core coverage of the first open source version released on August is ： Real time data synchronization 、 Development and  Fluent ETL.3  Months later, , We will release  Tapdata 1.5  edition , It is expected to add real-time data verification 、 Incremental data verification 、 Custom functions and aggregation operator scenarios support , At the same time, add the data source to  50  More than .

It is expected that  2022  year  11  month  30  Officially released on  2.0  The core capabilities of the version supplement include ：Any DB-To-API, Data directory 、 Data discovery 、 Data traceability , And increase the number of supported data sources to  80+. besides , We also have many capabilities that we want to explore and implement with all community developers in the evolution process , And problems solved through joint communication . some time ,Tapdata  The open source version will also be launched  Open API、 Streaming storage engine 、Open Metadata、Master Data  Wait for more pounds , Coming soon ！

Tapdata  The first batch of open source experience officers are under Limited Recruitment ！

To better listen to developers , Create elegance and ease of use 、 A fully functional real-time data platform ,Tapdata  Open source project experience officers are now publicly solicited , Rush to experience high-quality open source projects  Tapdata , Become the first members of the community , Meet more fellow developers , And  Tapdata  Work together to tap the potential of data , Get the latest information of the project at the first time .

As  Tapdata  The first batch of users in the community , You will be able to ：

• Make your data one step faster , feel  Data on Tap

• get  Tapdata  Open source  Issue、 Special priority of requirements

• Get the latest information from the community at the first time （ Including but not limited to the development plan 、 The core technology 、 Business scenarios, etc ）

• Participate in activities 、 Get the novice task of open source experience officer 、 Get the morning backpack 、 Tide card  T  Shirt and other good gifts

• Have the opportunity to be invited to join  Tapdata Committer Program, Become formal  Tapdata Committer

• Have the opportunity to directly participate in and influence  Tapdata  The future of

Link to the original text :【https://tapdata.net/tapdata-live-data-platform/news.html】