Tidb database characteristics summary

---

Preface

The project needs to use TIDB, First you need to deploy a TIDB Cluster environment , So I plan to learn TIDB database , Convenient for future use .

---

One 、TIDB Database introduction

1.1 Development stage of data management technology

1. Manual management stage ： No disk and other storage devices 、, Small amount of data , Directly managed by users .
2. File system phase ： There are disk and other storage devices , Store data on disk .
3. Database system phase ： Store the data in the database , Use database management tools to manage data uniformly .

1.2 Database classification

The database was proposed in the 1960s and 1970s , The purpose is to improve the efficiency of data management . In the order of development , Databases can be roughly divided into the following categories ：

1. Hierarchical and mesh data management system ： It can be understood as using pointers to represent the relationship between data .
2. Relational database （RDBMS）： Is the most commonly used database , It can be understood as using two-dimensional tables to represent and maintain the relationship between data . It promotes the miniaturization and popularization of the database .
3. A new generation of databases ： The new generation of database is mainly to solve the performance of relational database 、 Extensibility 、 Scalability and other issues , There are the following ：
   ORDBMS： Object oriented database technology , Such as PostGreSQL.
   NoSQL（Not only SQL）： It means more than just SQL, Advocate the use of non relational data storage , The general choice is to sacrifice the complex SQL Support and ACID Transaction in exchange for the ability of elastic expansion , Strong consistency is not usually guaranteed . Unstructured databases such as （1） The key stores the database ：Redis.（2） Column storage database ：HBase.（3） Document database ：MongoDB.（4） Graph database ：Neo4j.
   NewSQL： This kind of database not only has NoSQL The ability of the database to store and manage massive data , It also maintains the ACID and SQL Other characteristics , for example ：TiDB

thus it can be seen TIDB Belong to NewSQL Category , Is the latest database technology .

1.2 How to learn TiDB

This time right TIDB My study mainly includes the following parts , The order of arrangement is the learning route ：

• TIDB Characteristics and application scenarios of .
• TIDB Installation and deployment .
• TIDB Cluster deployment .
• TIDB Of SQL operation .
• TIDB Database migration .

Two 、TIDB Features and usage scenarios

2.1.MySQL Existing problems

When MySQL The amount of data in a single table exceeds 5000 when , The performance will drop dramatically , It is usually solved by dividing the database into tables . Although the sub database and sub table can make MySQL Stable and controllable performance , After splitting a single table into small tables , Be able to expand horizontally , Deploy to multiple servers , So as to improve the performance of the whole cluster QPS、TPS、Latency Wait for database service indicators . But if the server goes down , There is a risk of transaction inconsistency . After the sub table , Yes SQL There are certain restrictions on the sentence , Great discount on the functional requirements of the business side . After the sub table , Objects that need to be maintained grow exponentially （MySQL Number of instances 、 executable SQL Change quantity ）
To avoid MySQL Database problem , choose NewSQL,NewSQL The database has the following characteristics ：

• Unlimited level of expansion
• Distributed strong consistency , Ensure data 100% Security .
• Complete distributed transaction processing capability and ACID characteristic .

TiDBB yes NewSQL The technology center is a representative open source product .

2.2.TiDB Database features

TIDB yes Open source distributed relational database Is an online transaction oriented OLTP/ On line analytical processing OLAP The fusion database products , Realization One key horizontal expansion , Strong consistency of multi copy data security , Distributed transactions , real time OLAP And so on . Compatible with MySQL Agreement and Ecology , Easy to move , O & M cost polar .
Can be TIDB regard as MySQL Enhanced and distributed versions of
TIDB Database has Distributed strong consistency transactions 、 Online elastic horizontal expansion 、 High availability of fault self recovery 、 Across data centers And so on , It is an ideal database cluster and cloud database solution in the era of big data .
TIDB The design goal is 100% Of OLTP Scene and 80% Of OLAP scene , More complicated OLAP Analysis can be done by TiSpark Project to complete .

• Online transaction processing （OLTP）： Emphasize a large number of transaction operations in a short time , It emphasizes that it supports a large number of concurrent transaction operations in a short time （ Additions and deletions ） The ability of , Each operation involves a small amount of data . The emphasis is on the strong consistency of transactions （ bank transfer ）. for example “ Double 11 period , Hundreds of thousands of users may place orders in the same second , The background database should be able to be concurrent 、 Process these order requests in near real time ”
• Online transaction analysis （OLAP）： Prefer complex read-only queries , Read massive data for analysis and calculation , The query time is often very long . for example “ Double 11 ends , Taobao operators analyze and mine orders , Find out the market rules ”, In this analysis pit, you need to read all the historical orders for calculation , It takes tens of seconds or even minutes . On behalf of the product ：GreenPlum、TeraData,AnalyticDB

2.3TIDB Architecture features

Three core components ：TIDB Server, PD Server, TIKV Server. There are two additional components ：TiSpark,TIDB Operator.

• PD Server： Is the manager of the whole cluster , The main （1） Store metadata ,（2） Load balancing ,（3） Allocate globally unique transactions ID.
• TIDB Server： Responsible for receiving SQL request , adopt PD The metadata stored in the finds out where the data exists TiKV On , And with TiKV Interact , Returns the result to the client .
• TiKV Server： Responsible for actually storing data , It's essentially a KV Storage engine .
• TiSpark： Used to solve users' complex OLAP Query the components of the requirements .
• TiDB Operator： Components used to facilitate cloud deployment .

2.4 TiDB Core features

1. Highly compatible MySQL： in the majority of cases , You can start from... Without modifying the code MySQL Easily move to TIDB, After sub database and sub table MySQL The cluster can still pass through TIDB Tools for real-time migration .
2. Distributed transaction support ：100% Support ACID Business .
3. "One-stop" work style HTAP Solution ： Mixed transaction processing and analysis processing .
4. Cloud native SQL database ：TIDB It is designed for the cloud , Support public 、 Private and hybrid clouds work together TIDB Operator The project can realize automatic operation and maintenance 、 To deploy 、 Configuration and maintenance are very simple .
5. Horizontal elastic expansion ：（ Storage and computing power ） You can simply add new nodes TIDB Horizontal expansion of , Expand throughput or storage on demand , Easy to deal with high concurrency 、 Massive data scenarios .
6. Real financial grade high reusability ： Compared with the traditional master-slave （M-S） Copy scheme , be based on Raft The majority election agreement of can provide financial grade 100% Strong data consistency guarantee , And without losing most copies , It can realize automatic fault recovery （anto-failover）, No manual access is required .

2.4.1 Horizontal expansibility

Horizontal expansion is TIDB One of the characteristics of , The horizontal expansion here includes two aspects ： Computing power and storage power .
TIDB server Responsible for handling SQL request , As the business grows , You can simply add TIDB Server To improve the overall processing capacity , Provide higher throughput .
TiKV Responsible for storing data , As the volume of data grows , You can deploy more TiKV server Nodes solve data Scale The problem of .
PD Will be in TiKV Between nodes Region Schedule for units , Migrate some data to new nodes .
So in the early days of business , Only a few service instances can be deployed （ It is recommended to deploy at least three TiKV, Three PD, Two TIDB）, As the business grows , Add... As required TiKV and TIDB example .

2.4.2 High availability

High availability ： High availability is TIDB Another big feature of ,TIDB/TiKV/PD All three components can tolerate partial instance failure , Does not affect the availability of the entire cluster .
1、TIDB：TIDB It's stateless , Visit the first one 、 The second or third is the same , You can deploy a few more , The front end provides external services through load balancing components , After a single instance fails , You can restart this instance or deploy a new one .
2、PD：PD It's a cluster , adopt Raft The protocol keeps the data consistent , When a single instance fails , If this example is not Raft Of leader, Then the service is completely unaffected , If it is Raft Of leader, Then a new one will be selected Raft leader, Automatic reply service ,PD In the election process , Unable to provide external services , This time is about three seconds . It is recommended to deploy at least three PD example , After a single instance fails , Restart this instance or add a new one .
3、TiKV：TiKV It's a cluster , through Raft The protocol keeps the data consistent （ The number of copies can be configured , The default is to save three copies ）, adopt PD Do load balancing scheduling , When a single node fails , It will affect all stored on this node Region. about Region Medium Leader node , Service will be interrupted , Waiting for re-election . about Region Medium Follower node , It will not affect the service . When a TiKV Node failure , And can't reply within a period of time （ Default 30min）,PD The data on it will be migrated to other TiKV Node .

2.5 TIDB Storage and computing power

TIDB The storage and calculation of are respectively controlled by TiKV and TIDB Server complete .
Storage capacity TiKV Server Usually 3+ Of ,TIDB Each piece of data defaults to three copies .TiKV The data format of cluster storage is KV Of , stay TIDB in , Instead of storing data directly in HDD/SSD On , But through RocksDB Realized TB Level local storage scheme , Make a point ：RocksDB and HBASE equally , It's all through LSM Tree as storage scheme , Avoided B+ A large number of random reads and writes caused by the expansion of tree leaf nodes , Thus, the overall throughput is improved .

Ability to calculate TIDB Server Itself is stateless , It means that when computing power becomes a bottleneck , The machine can be directly expanded , Transparent to users . Theoretically TIDB Server There is no upper limit to the number of .

2.6 TIDB Feature summary

TIDB As a new generation NewSQL database , It has gradually gained a firm foothold in the field of database , Combined with the Etcd/MySQL/HDFS/HBase/Spark And other technologies , With TIDB Large scale promotion of , Will gradually weaken OLTP/OLAP The boundaries of , And simplify the current redundancy ETL technological process , Cause a new round of technology wave . blunt ,TIDB Future period .

3 TIDB Deploy

TIDB The production environment affects the system and memory 、CPU There are requirements , If not, it cannot be installed , So in V4.0 Previously recommended docker compose Deployment ）（ If you just want to test TIDB, Experience TIDB Characteristics of , Or for the development environment , You can use Docker Compose Rapidly deploy locally TIDB colony , This deployment method is not applicable to the production environment ）. But in the V4.0 And then ,TIDB Developed its own cluster deployment management tool ——TiUP,TiUP It is the present. TIDB Recommended cluster deployment management tools , So in V4.0 after , I recommend you to use TiUP For deployment testing or actual deployment .
TiUP yes TIDB4.0 Cluster operation and maintenance tools introduced in version , adopt TiUP Can be done TIDB Daily maintenance work of , Including deployment 、 start-up 、 close 、 The destruction 、 Elastic expansion and upgrading TIDB colony , And management TIDB Cluster parameters .

My Tencent cloud server is linux System CPU by amd64 framework ,2 nucleus 4g,6m bandwidth , But when a single machine is deployed to simulate a production environment cluster , Club card , It is impossible to , Therefore, the author uses virtual machine for deployment , Open up a lot of memory for virtual machines , Finally, after solving some pits , Successful deployment , The next blog will explain in detail the single machine simulation deployment of production environment clusters , Encountered pits and solutions .

---

summary

This blog is about TIDB Introduced , And the development and classification of the database , And the currently widely used MySQL The bottleneck of database is described .TIDB Is an open source distributed relational database , yes NewSQL A representative of .TIDB It has many excellent characteristics , for example ： Achieve one key horizontal scaling , Strong consistency of multi copy data security , Distributed transactions , real time OLAP And so on . In a word ：TIDB Future period , The next blog will introduce the method of deploying production environment clusters by single machine simulation , Pits encountered and solutions .