HTAP in depth exploration Guide

This guide describes how to further explore and use TiDB Online transaction and online analytical processing (Hybrid Transactional and Analytical Processing, HTAP) function .

Be careful

If you are right about TiDB HTAP I don't know much about the functions , Hope to have a quick trial experience , see also Quick start HTAP.

Get to know... Quickly TiDB stay HTAP The architecture under the scenario is similar to HTAP The applicable scenarios of , It is recommended to watch the following training video first （ Duration 15 minute ）. Note that this video is for reference only , For more information HTAP Related content , Please refer to the documentation below .

HTAP Applicable scenario

TiDB HTAP It can meet the increasing production demand of enterprises' massive data 、 Reduce the risk cost of operation and maintenance 、 Seamlessly stitch with the existing big data stack , So as to realize the real-time realization of the value of data assets .

Here are three kinds of HTAP Typical applicable scenarios ：

• Mixed load scenarios

  When will TiDB When applied to the mixed load scenario of online real-time analysis and processing , Developers only need to provide an entry ,TiDB Different processing engines will be automatically selected according to the business type .

• Real time streaming scenarios

  When will TiDB When applied to real-time stream processing scenarios ,TiDB It can ensure that the data flowing into the system continuously can be checked in real time , At the same time, high concurrent data service and BI Inquire about .

• Data hub scenario

  When will TiDB When applied to data hub scenarios ,TiDB As a data hub, it can seamlessly connect the data business layer and the data warehouse layer , Meet the needs of different businesses .

If you want to know more about TiDB HTAP Scene information , see also  PingCAP On the official website HTAP The blog of .

HTAP framework

stay TiDB in , Row storage engine for online transaction processing  TiKV  And the column storage engine for real-time analysis scenarios  TiFlash  At the same time , Automatic synchronization , Maintain strong consistency .

More architecture information , Please refer to  TiDB HTAP Morphological architecture .

HTAP Environmental preparation

In depth exploration TiDB HTAP Before function , Please deploy according to your data scenario TiDB And the corresponding data analysis engine . Big data scenario (100 T) Next , Recommended TiFlash MPP As HTAP The main scheme of ,TiSpark As a supplementary scheme .

• TiFlash

  • If already deployed TiDB Cluster but not yet deployed TiFlash node , see also Capacity expansion TiFlash node The steps in are in the existing TiDB Add in cluster TiFlash node .

  • If not already deployed TiDB colony , Please use  TiUP Deploy TiDB colony , And on the basis of containing the minimum Topology , meanwhile increase TiFlash topology .

  • In deciding how to choose TiFlash When the number of nodes , Consider the following business scenarios ：

    • If the business scenario is based on OLTP Mainly , Do lightweight Ad hoc OLAP Calculation , Usually deploy 1 One or more TiFlash Nodes will have a significant acceleration effect .
    • When OLTP Data throughput is important to nodes I/O When there is no obvious pressure , Every TiFlash Nodes will use more resources for computing , such TiFlash Clusters can achieve approximately linear scalability .TiFlash The number of nodes should be adjusted according to the expected performance and response time .
    • When OLTP When the data throughput is high （ For example, write or update more than ten million lines / Hours ）, Due to the limited writing capacity of the network and physical disks , Inside TiKV And TiFlash Between I/O Will become the main bottleneck , It is also easy to generate read and write hotspots . here TiFlash Number of nodes and OLAP The amount of calculation has a complex nonlinear relationship , The number of nodes needs to be adjusted according to the specific system status .

• TiSpark

  • If your business needs to be based on Spark Analyze , Please deploy TiSpark. Specific steps , see also  TiSpark User guide .

HTAP Data preparation

TiFlash Data will not be synchronized automatically after deployment , You need to specify that you want to sync to TiFlash Data sheet for . After designation ,TiDB The corresponding TiFlash copy .

• If TiDB There is no data in the cluster , Please migrate the data to TiDB. Please refer to Data migration .
• If TiDB The cluster already has data synchronized from the upstream ,TiFlash Data will not be synchronized automatically after deployment , You need to manually specify the tables that need to be synchronized , Please refer to Use TiFlash.

HTAP Data processing

Use TiDB when , You just type in SQL Statement to query or write requirements . For creating TiFlash Copy of the table ,TiDB It will rely on the front-end optimizer to freely select the optimal execution method .

Be careful

TiFlash Of MPP Mode is on by default . When executed SQL When the sentence is ,TiDB The optimizer will automatically determine and select whether to use MPP Mode execution .

• If you need to shut down MPP Pattern , Please change the system variable  tidb_allow_mpp  Is set to OFF.
• If mandatory TiFlash Of MPP Schema execution query , Please change the system variable  tidb_allow_mpp  and  tidb_enforce_mpp  Is set to ON.
• To view TiDB Whether to select with MPP Mode execution , You can adopt EXPLAIN Statement to view the specific query execution plan . If EXPLAIN Statement ExchangeSender and ExchangeReceiver operator , indicate MPP In force .

HTAP Performance monitoring

stay TiDB In the course of using , You can select the following methods to monitor TiDB Cluster operation and view performance data .

• TiDB Dashboard： View the overall operation overview of the cluster , Analyze the distribution and trend of cluster read / write traffic , Learn more about the time-consuming SQL Statement execution information .
• The monitoring system (Prometheus & Grafana)： see TiDB Cluster components （ Include PD、TiDB、TiKV、TiFlash、TiCDC、Node_exporter） Relevant monitoring parameters .

To view TiDB and TiFlash Cluster alarm rules and processing methods , Please refer to  TiDB Cluster alarm rules and  TiFlash Alarm rules .

HTAP Troubleshooting

In the use of TiDB If you encounter problems in the process of , Please refer to the following documents ：

• Analyze slow queries
• Locate queries that consume more system resources
• TiDB Dealing with hot issues
• TiDB Cluster fault diagnosis
• TiFlash common problem

besides , You can  Github Issues  Create a new one Issue Feedback questions , Or in  AskTUG  Submit your question .