Tidb hybrid deployment topology

In this paper, TiDB Clustered TiKV and TiDB Hybrid deployment topology and main parameters . Common scenarios are , The deployer is multi-channel CPU processor , There's plenty of memory , To improve the utilization of physical machine resources , Single machine multi instance deployment , namely TiDB、TiKV adopt numa To nuclear , Isolation CPU resources .PD and Prometheus Mixed deployment , But the data directories of both need to use independent file systems .

Topology information

example	Number	Physical machine configuration	IP	To configure
TiDB	6	32 VCore 64GB	10.0.1.1 10.0.1.2 10.0.1.3	To configure numa Binding operation
PD	3	16 VCore 32 GB	10.0.1.4 10.0.1.5 10.0.1.6	To configure location_labels Parameters
TiKV	6	32 VCore 64GB	10.0.1.7 10.0.1.8 10.0.1.9	1. Distinguish instance level port、status_port; 2. Configure global parameters readpool、storage as well as raftstore; 3. Configure instance level host Dimensional labels; 4. To configure numa Binding operation
Monitoring & Grafana	1	4 VCore 8GB * 1 500GB (ssd)	10.0.1.10	The default configuration

Topology template

Simple mixed part configuration template

Detailed mix configuration template

above TiDB In the cluster topology file , See adopt TiUP Deploy TiDB Cluster topology file configuration .

Introduction to key parameters of hybrid deployment

This section introduces the key parameters of single machine multi instance , It is mainly used for TiDB、TiKV Single machine multi instance deployment scenario . You need to follow the calculation formula provided , Fill the results in the configuration template of the previous step .

• TiKV Make configuration optimization

  • readpool Thread pool adaptation , To configure  readpool.unified.max-thread-count  Parameters can make  readpool.storage  and  readpool.coprocessor  Share a unified route pool , At the same time, adaptive switches should be set separately .

    • Turn on  readpool.storage  and  readpool.coprocessor：

      readpool.storage.use-unified-pool: true readpool.coprocessor.use-unified-pool: true

    • The calculation formula is as follows ：

      readpool.unified.max-thread-count = cores * 0.8 / TiKV Number

  • storage CF (all RocksDB column families) Memory adaptation , To configure  storage.block-cache.capacity  Parameters can be implemented CF Automatically balance memory usage .

    • storage.block-cache  Default on CF The adaptive , There is no need to modify .

      storage.block-cache.shared: true

    • The calculation formula is as follows ：

      storage.block-cache.capacity = (MEM_TOTAL * 0.5 / TiKV Number of instances )

  • If more than one TiKV Instances are deployed on the same physical disk , Need to be in tikv Add in configuration capacity Parameters ：

    raftstore.capacity = Total disk capacity / TiKV Number of instances

• label Scheduling configuration

  Due to the single machine multi instance deployment TiKV, To avoid physical downtime leading to Region Group Default 3 Replica 2 Copy lost , Problems that make the cluster unavailable , Can pass label To achieve PD Intelligent scheduling , Ensure that there are many machines on the same machine TiKV Instances will not appear Region Group Only 2 The situation of the copy .

  • TiKV To configure

    The same physical machine has the same configuration host Level label Information ：

    config: server.labels: host: tikv1

  • PD To configure

    PD Need configuration labels Type to identify and schedule Region：

    pd: replication.location-labels: ["host"]

• numa_node  To nuclear

  • Configure the corresponding... In the instance parameter module  numa_node  Parameters , And add the corresponding physical CPU The number of nuclear ;

  • numa Before binding core for use , Confirm that it has been installed numactl Tools , And the physical machine corresponding to the physical machine CPU After , Then configure the parameters ;

  • numa_node  This configuration parameter is similar to  numactl --membind  Configuration correspondence .

Be careful

• When editing the profile template , Pay attention to modifying the necessary parameters 、IP、 Port and directory .
• Of each component deploy_dir, By default global Medium  <deploy_dir>/<components_name>-<port>. for example tidb Port designation 4001, be deploy_dir The default is '/tidb-deploy/tidb-4001'. therefore , When specifying a non default port in a multi instance scenario , There is no need to specify the directory again .
• There is no need to manually create  tidb  user ,TiUP cluster The component will automatically create the user on the deployment host . You can customize the user , It can also be consistent with the users of the central control computer .
• If the deployment directory is configured as a relative path , It will be deployed in the user's home directory .