Open source to the world (Part 2): the power of open source from the evolution of database technology BDTC 2021

This article is based on  PingCAP Fanruohan, senior vice president, is in BDTC 2021 Keynote speech of ：“ Open source to the world ” Sort out , From the perspective of collaboration mode and technology evolution , Shared “ Open source ” and “ Globalization ” They're connected to each other , An inseparable relationship , Divided into two parts , The first part introduces how open source builds the stage of globalization , The theme of this article is ： Looking at the power of open source from the evolution of database technology

We think , The driving forces of data technology evolution can be summed up , It mainly includes three aspects ： Theoretical basis promotes software innovation 、 The infrastructure guarantees the realization of software capabilities 、 Business requirements really polish the continuous engineering of technology 、 Commercialization , It is the real product “ place ”.

Database evolution history —— Basic theory drives

In terms of time and function , We have divided the data ecology , It's about SQL ecology , Big data Ecology ,NoSQL ecology ,NewSQL ecology , as well as SQL Cloud ecology of . The evolution of each Ecology , It is inseparable from the development of basic theory .

1970 year IBM Relational database theory Relational Model Include System R The advent of prototype products , by Oracle、DB2、MicroSoft SQL Server The birth of these commercial databases laid the foundation . Then MySQL、PostgreSQL In the form of open source, it has achieved rapid development and the most extensive application in the world .

2003 year - 2006 year , Google Troika GFS,MapReduce,BigTable The publication of the paper , Laid the theoretical foundation for large-scale distributed storage systems in the industry . It is very popular nowadays Hadoop、Spark、MongoDB、Hbase And so on are all based on these theories . You can see , These data products are developed and expanded in the open source mode . Because the iteration speed of closed source mode is slow , The unit cost is high , It has been unable to meet the needs of a large number of users .

2012 year - 2014 year , still Google Published Spanner and F1, And Stanford University Raft The paper , To promote the NewSQL The development of database .PingCAP Of TiDB, It is also the product realization of these theoretical bases , And continue to innovate on this basis .

Database evolution history —— Driven by business innovation

Let's see how to understand what I just said “ place ”, Overall speaking , Business requirements are reflected in the following three aspects ：

One is  “ Transaction features ” , That is to say ACID Atomicity 、 Uniformity 、 Isolation, 、 persistence . Generally speaking , Process digitization 、 Online business is a serious business , Like finance 、 Telecommunications and other services , And enterprise class ERP、CRM, All require reliable transaction features .

Two is  “ Data scale ” , It is mainly reflected in the explosive growth of massive data brought by the Internet , Whether user behavior is fully Internet-based 、 Or the data collection brought by mobile devices is extremely rich , Or is the creation of content itself causing a huge surge in data , From text to picture 、 Animation 、 Short video 、 Long video 、 Game to the recent hot metauniverse , Are all factors of data scale growth , Under the extreme impetus of the epidemic , The digital transformation of various industries has given birth to a new round of data growth .

The third is  “ Processing delay ” , In today's mobile Internet and digitalization , The pursuit of user experience goes up ,ToC The business wants faster service response , So as to compete for the fragment time of users , Compete for business opportunities ,ToB Business also needs more rapid business response to data processing , More real-time data analysis and more agile operational decisions .

These three factors have different development in different periods , There are different combinations , The development of data technology has been continuously promoted and implemented . Different database ecosystems , It is the result of different combinations of business drivers .

In the information age, the digital foundation is weak , It mainly solves the accuracy and efficiency of key businesses , More serious business with small amount of data , Highly transactional requirements for data , And the data structure is stable 、 The rules are clear , The amount of data is limited , This type of demand relationship SQL Stand alone database ecology can meet , What is required is efficiency and stability .

2000 It entered the big data era around , After the long-term development of informatization , The data has accumulated a lot , New data is also increasing at an unprecedented volume and speed , The stand-alone relational database is showing signs of difficulty and obsolescence . In order to store and analyze massive data , Especially the data accumulated offline , All kinds of high efficiency 、 Telescopic 、 Big data processing platforms that can be deployed on low-cost hardware have emerged one after another .

Then the early days of the Internet era , The content and online behavior of users are extremely rich , But at that time, it was mainly massive unstructured data storage （ video / Audio / Image & Text / Social relationships, etc ）、 But the data scale is huge , Concurrent traffic is required 、 And compete for traffic , Quick response to user access 、 The need to provide a low latency user experience drives NoSQL The development of ecology . Because the early Internet business was not for profit , The data to be processed is more the browsing records of users on the Internet , Social relationships, etc , Therefore, there are not so high requirements for transaction characteristics .

Enter the era of mobile Internet , With the rapid growth of data volume , While ensuring a good user experience, the business also needs to complete transactions and realize cash , Business agility requires the system to respond quickly to business changes and data growth , It also requires highly reliable support for massive transactions 、 Payment and other serious matters . You can see , During this period, the three elements of the business driving force came into view . Some enterprises are still passing SQL The transition mode of ecological cloudization to meet , But we also see in practice , When the user's data volume, especially the data update, exceeds a certain range , Native distributed NewSQL Is the choice of advanced architecture . In addition, data technology has entered the stage of full cloud service , The differences in architecture are even more obvious .

meanwhile , Real time insight requires data decisions from T+1 towards T+0 upgrade , Even second level and millisecond level analysis response , Real time aggregation of multi-source data 、 Dynamic updating and flexible computing are emerging requirements , Gradually, the boundary between transactional computing and analytical computing becomes more and more blurred , The technological innovation of database and big data will continue to integrate .

Database evolution history —— Infrastructure driven

Last , Hardware is the cornerstone of software , The development of data technology is inseparable from the development of infrastructure .

From mainframe to X86 From servers to cloud computing , Infrastructure deployment enables from “ year ” To “ month ” To “ Japan ” To “ second ” A disruptive change , Resources from proprietary 、 Closed to on demand start 、 Elastic expansion . The cloud primary era once again expands the scale of resources , The granularity of resources is reduced ,API turn 、 Microservicing further brings the business online 、 The update speed is pushed to the second level .

future , The design of resource separation will release greater power in the cloud . The timeline of the above database development is only drawn to NewSQL, In fact, data technology is still evolving . Believe that in this process , Open source will bring more and more value . The core behind all cloud products now comes from open source , The driving force of innovation also comes from open source . The following is in 2021 PingCAP DevCon At the conference , Dongxu put forward a bold assumption ： The age of cloud origin , All that can be separated will be separated , The scale effect controls everything . This separation includes the separation of storage and computation 、 What is more extreme is the separation of storage for different purposes , Business Computing can be further separated from distributed computing and transactional computing . Continue to optimize the scale effect and resource efficiency to the extreme , For users , Just focus on the business itself , The rest is left to the cloud database .

TiDB The Enlightenment of product iteration

Under the action of these three driving forces , We can sum up TiDB In the past 6 The product iteration in the middle of the year .

TiDB The biggest advantage of the product is the openness of technology , Open architecture means that more connections can be generated , More connections means faster iterations 、 More possibilities .

TiDB Our original intention is to provide a native distributed and well supported OLTP Transaction database , Let's have DBA No longer stay up late because of the huge amount of data .TiDB 1.0 and 2.0 The solution is this problem . Later, with the real-time demand brought by digitalization , One stack HTAP Become the direction of our efforts , With this year TiFlash MPP Release , We have achieved comprehensive HTAP Ability .

As a cloud native distributed database , We launched this year TiDB Cloud, And free for developers to try Developer Tier, Users can go to Amazon Web Services Free running on TiDB Cluster one year .TiDB Cloud Responsible for infrastructure management 、 Cluster deployment 、 Backup management and other background database management , So that developers can focus on creating excellent applications , Achieve second level switching . All these are based on the iteration speed and innovation power brought by open source .

Last use RedHat CEO Paul Cormier In a recent TV interview, I said a sentence as a summary ：Open source software is the heart of the technology behind cloud computing.

Is it open source , Do you want to open source , Is it necessary to take open source as an external driving force for the company's continuous innovation , I think it is a problem that every basic software company can think deeply .