China's technology goes to sea, tidb database's overseas exploration road | interview with excellent technical team

author | Xin Xiaoliang

Interviewing guests | Liu Song 、 Chen Chen 、 Pei Liquan

According to the PingCAP Introduce , At present, their TiDB Database products have served many overseas giant enterprises , Internet coverage 、 Technology 、 Finance 、 Games and other industries . Developers familiar with databases know , With basic software “ troika ” How important is one's database to an enterprise .

PingCAP From the beginning of its establishment, it has adhered to internationalization , How they succeeded in going to sea , In the overseas market with a large number of giants, we have a successful foothold .TiDB What technical optimization has been done , What are the characteristics of overseas business , What does it do for these scenarios ？ What challenges have they encountered in their overseas business ？ This time ,InfoQ In depth interview PingCAP The core members of the team , understand TiDB Those things about database overseas exploration , I hope it can inspire you .

1 TiDB Cloud platform evolution

For those with global business PingCAP Come on , Database products TiDB Cloud adaptation is a required option . Especially for overseas customers , from DB To DBaaS, Only cloud services can break through the geographical restrictions , Provide unlimited computing power .

However, going to the cloud is not just as simple as changing the underlying resources to the cloud , technical , To achieve cost reduction and efficiency increase 、 Operation and maintenance automation 、 Multi tenant management ; Data security should be considered in compliance 、 Regulatory rules ; Commercially , Pricing model 、 Commercialization strategy and other factors need to be taken into account .

Let's talk about it from a technical point of view TiDB Deploy locally to TiDB Cloud （DBaaS） Problems solved in the process of evolution .

The first is the separate architecture design . In the past TiDB Use TiDB + TiKV Coprocessing engine , The boundary between storage and calculation is fuzzy , Deploy... Locally （ On-Premise ) The situation of , If you need to increase storage capacity , You need to add storage nodes , Due to hardware limitations , Except disk ,CPU And the network bandwidth will also increase synchronously , It is difficult to deal with scenarios with different load rates , It is also easy to waste resources .

After moving to the cloud , Everything becomes different . With AWS Block storage device EBS For example , In especial GP3 series , It can run on different machines , Performance is very good , The integration of cloud native is also good . In order to take advantage of GP3 These characteristics of ,TiDB Move the calculated and stored boundaries down , From the original TiKV To the storage , Into the present TiDB、TiKV Most of can be computing units .

In addition to computing storage separation , Components with different roles also have different requirements for hardware resources , Select the storage service type as needed 、 Elastic computing resources 、 Serverless computation 、 Operation and maintenance automation will become a cost saving scheme .

In general, the most important thing cloud native technology solves is the cost problem . Apart from the cost , The cloud Security It's also the top priority . The security system on the cloud is different from that under the cloud , Under the cloud, we only need to consider RBAC Permissions inside the database , On the cloud, there should be a complete set of sound security system from network to storage , The key is to use the security mechanism provided by the cloud itself , Such as key management 、 Rules etc. .

Finally back to business , Provide services for overseas customers on the cloud , In addition to technology, another prerequisite is compliance . The main line of ecological integration on the cloud is to follow the data , Data upstream 、 The downstream 、 Management and control are the three most important parts . With TiDB For example , Its upstream is MySQL、S3 Data files in ; Downstream only needs to support Kafka Or other message queuing services ; Finally, for overseas users on the cloud , Compared with database manufacturers to do control , They prefer to work with DataDog、Confluent Wait for the platform to get through .

2 from TiDB Look at the sea mode of domestic technology

When Chinese technology goes to sea, it is easy to encounter trust 、 regulatory 、 Various problems such as geographical location . Take the European Union for example , Although the country is relatively small , But the number is very large , and GDPR Very strict , Personal data are not allowed to leave the country ; Another example , Indonesia consists of many islands , In response to natural disasters , It requires data to be archived hundreds of kilometers away , In addition, Japan is a disaster prone country , Frequent tsunamis 、 The earthquake , Cross regional backup will also be required , For example, public cloud manufacturers usually put some in Tokyo , Osaka put some . therefore TiDB Disaster recovery of cross regional data must also be supported . Besides , Overseas labor costs are becoming more and more expensive , Therefore, some regions will require all technical facilities to be managed with code , Yes Infrastructure as Code namely IaC It will be a big challenge , At the same time, the requirements for automatic operation and maintenance are also very high .

In addition to basic technology TiDB There are several key factors for smooth sailing . The first is the cloud , Only cloud services can break through the geographical restrictions , As mentioned earlier , No more details here .TiDB After the cloud platform is completed , The second important thing is to lift the threshold of trust . Have to admit , At this point , Open source is a good stepping stone . Chinese technology companies serve overseas customers , If you don't use open source technology , Doubt will be heightened , Especially the infrastructure of database , In the words of experts ：“ The database is like heart bypass surgery , It's a deadly thing .”TiDB The activity of the open source community also helps a lot in this regard , It solves the problem of trust threshold , At the same time, it also saves a lot of trouble in advertising and foreign markets .

The third is to be sensitive to new technology trends , With TiDB For example , Open source technology iterates quickly , Can be compared with other technologies such as Flink、Spark Merge together . Many overseas enterprises look very far forward to technology , The evolution of technology must keep up with the new forms of customers in order to ensure that it does not fall behind . Looking back at China , Our cloud vendors do not have particularly advanced technologies in the architecture , The situation of homogeneous competition and volume continues , There are a lot of tools , But the model has no obvious advantages .

Overseas users are basically in the cloud , Many are used in the cloud MySQL Or in the cloud RDS, This is right TiDB It is also a natural advantage .TiDB Our approach is to give priority to the head Internet enterprises on the overseas public cloud , Public cloud solves the problem of data security , Internet companies have solved the problem of volume . Whether the other party is using MySQL Or other traditional databases , There will be new demands when encountering bottlenecks , Happen to happen TiDB Backwards compatible MySQL. meanwhile ,TiDB The form of open source also plays a key role .

3 Serving global scenarios ,TiDB Overseas exploration and practice

about TiDB Come on , There are also obvious differences in the needs of overseas customers in different regions , At present, there are two main branches of overseas cloud customers , Most of them used RDS, Among them MySQL Of RDS Most commonly , When there is an amount of data that cannot be processed, there are two options . The first is full hosting , That is, all technical things are ignored , Leave all services to TiDB, This is the same. TiDB Cloud Do the things , This branch is currently dominated by Japanese customers ; Another part is to prevent cloud vendors from locking , For example, most customers in the United States hope not to be bound by a cloud vendor , It is even possible to prepare for local deployment due to cost considerations in the future , Then they will choose a database solution that meets both cloud and local support . Users in North America are also interested in fully hosted TiDB Cloud Showed a lot of attention , last year 11 month TiDB Cloud Free version for developers Dev Tier There will be a high number of visits and trial users .

Of course, there is also the need for some architecture renovation , For example, large technology companies tend to be more active in seeking more advanced solutions , To expect better business returns .

Take a famous social networking company in North America A For example ,A The company has a large number of users ,HBase The scale has reached 10PB.A When the company was founded, it was NoSQL The more popular stage , So a lot of NoSQL Service for , In fact, many businesses need extensible relational database , But there was no better choice . Later, with the development of , They slowly found out NoSQL It's hard to meet the needs of the business , A lot of custom development is needed , Ready to find the next database solution , It took half a year , I saw 15 A system , It can be seen from here that ,A The company is still strict in the selection of database . Finally they chose TiDB, The main reason is TiDB No need for much custom development , Emancipate the mind , High maturity 、 And low maintenance cost .A After the evaluation of the company , Begin to gradually put HBase towards TiDB transfer .

In fact, it can be seen from the choice of the company's technology stack , Although there are not enough typical customers at present , however NoSQL Replacement will be a trend ： Many users choose NoSQL Not necessarily to unbind relational features , Instead, they may be forced to sacrifice the features of relational databases just for scalability .

There's another case ,B The company previously spent nearly ten million dollars on 300 Multiple machines Aurora, But the effect is not so good .300 The upgrading of multiple machines will be very heavy for them ,Aurora Nor can it meet the current business needs . Besides ,B The company is doing Data Service, Previous SQL The scheme is also poor for the current application development experience . So based on these things ,B The company wants to use TiDB To support , Support 100 More than the QPS.B Companies choose TiDB The reason is that TiDB Better observability , At the same time, the architecture can be simplified , This is very close to their next generation cloud native architecture .B The company's next-generation cloud native architecture hopes to deploy one set on each public cloud Kubernetes, Give Way TiDB across Kubernetes Services provided in , On the whole, it's a TiDB colony .

The Japanese market has its own unique scene , After research TiDB It is found that its main characteristics are as follows , First of all, the pain point of the difficulty of traditional database expansion is more prominent , The demand of real-time business makes traditional companies consider gradually trying to replace Oracle Such as the database , upgrade MySQL Database expansion capability , And improve the ability of real-time analysis ; second , The COVID-19 has a certain boosting effect on the entire digital industry in Japan , Although the total population is not large , But the scenario tends to be more and more high-frequency business , This is a perfect situation for TiDB Develop ; Third 、 Due to labor cost and various reasons , Japan lacks experienced SRE Engineer and DBA（ Database administrator ）. meanwhile TiDB Find out , Based on these factors TiDB What can be done plus the impact of the open source community ,TiDB The heat in Japan is still relatively high .

TiDB The plan in Japan is to start with the game 、 Start with the Internet industry , Then slowly turn to traditional industries 、 Financial industry . On the one hand, the game industry has a natural cycle , Secondly, there are high-frequency scenes in the Internet industry , There is a growing demand for large amounts of data , Another reason is that they generally have a high acceptance of new technologies , Less concern about things other than product technology , Just look at the product capability and success , That's exactly what it is. TiDB Needed .

But unlike North America , Japanese customers prefer to use hosted services , This allows them to focus more on the business itself . Take the game industry for example , Japanese games are characterized by a long life cycle , A game has been running for decades , There are many loyal fans , Users also have good paying habits , So game payment is their most common scene, such as high-frequency trading 、 timeliness 、 Transaction integrity and other requirements are very high , In addition, they always have very stable users , These users attach great importance to the stability of the game and the maintenance period caused by game upgrades and other reasons . Therefore, Japanese game companies will also focus on game production , For example, collect the sound of the game . It's not that they don't pay attention to technology , Just because of labor costs , Relatively inexperienced SRE Engineer and DBA（ Database administrator ）, These are great challenges for cloud databases .

Take a game company C For example ,C The company's annual game revenue is about 2-3 Billion dollars , They develop ten games a year , But the development team has only 10 personal , If it is subdivided into database and cloud development teams , There are fewer people , Based on non-native distributed solutions, such as MySQL Sharding The complexity of requires more engineers , For small teams, it will significantly slow down the business .TiDB Cloud This kind of trusteeship is obviously more appropriate . meanwhile ,TiDB In real-time feedback 、 High frequency trading 、 timeliness 、 Integrity, etc. the above also really meet the requirements of the other party .

Another characteristic is the financial industry , Non cash payment in the Japanese market is not developing so fast as in China , They had a rapid growth process a few years ago , After analysis, it is found that there are mainly the following reasons , The first is the rapid growth of e-commerce , Second, the development of smart phones and electronic payment , The third is the influx of foreign tourists , Electronic payment is the most convenient payment channel for them , These surge in users 、 Real time online transactions make the local payment company database unable to host .

Take one of the financial companies D For example , Their architecture is very simple , Mainly including payment 、 user 、 Merchant 、 wallet 、 Risk control 、 Activities and other modules . They are from 1500 The number of users has increased to 4000 In the process of ten thousand , The original Aurora The database cannot handle the demand of real-time online , You have chosen to migrate to TiDB.D The company's demand is to carry 1000 TPS, Because their architecture is a write read nature , The existing Aurora It is difficult for the database to break through this bottleneck ,TiDB It is a multi write horizontal expansion Architecture , Migrate to TiDB after ,TPS It's also three times as easy as before . besides ,TiDB And solved the previous Aurora Performance issues while providing better recoverability .

Finally, a brief summary , In addition to basic technology 、 Performance and other key product features can meet their own scenarios and needs ,TiDB There are several other factors that attract overseas customers to choose , First of all TiDB It's open source. , There is an active open source community ; Second, not as a data container , The customer's data will still be stored in AWS、GCP Wait for the public cloud ; The third is remote support , You don't have to rely on local technical teams , It can effectively solve the problem of expensive human resources in some areas .

4 The future trend of database

Into the cloud 2.0 Time , Before RBS（ long-range Blob Storage ） Groups covered , Basically, there are already leading customers to upgrade . One of them is cross cloud 、 Cloud native ; The second is to adopt a completely managed cloud neutral approach DBaaS such , such as TiDB Cloud. In addition, today's distributed database is not just a database , It has gradually become a Data Platform （ Data platform ）, At the same time, it also handles a lot of big data work, such as HTAP Instead of real-time analysis of big data , According to the experts interviewed , Eventually, it will evolve into a cloud with the help of open source Data Ecosystem（ Data ecosystem ）.

The current database , Not really native to the cloud, such as Serverless To design . The real next generation database , It should have the following characteristics . First, we can get more and more flexible resources from the cloud , Saving can be done at will , Now common databases are deployed in the cloud , In theory, you need to 1000 Computing nodes , Fewer storage nodes , At night it's the opposite ; Second, the bottom layer of the future database, such as the storage layer , Be able to achieve deeper optimization in the cloud , Supporting image SnowFlake This can dynamically call cloud resources , According to different types of requests, decide what kind of storage logic to choose at the bottom . This is the real cloud native in the future , It's still on the way , But the expectation can be realized in the future .

Introduction of interview guests ：

Liu Song ：PingCAP Vice President, , I was in Oracle, Alibaba cloud holds positions such as technology management

Chen Chen ：PingCAP Japan branch Chief technical officer , Once worked in Alibaba cloud ,IBM As a technical development consultant

Pei Liquan ：PingCAP North American branch Chief technical advisor , I was in Pinterest Technical operation and maintenance and other positions