Ticdc introduction

TiCDC Through the pull TiKV Change log , Realization TiDB Incremental data synchronization tool , It has the ability to restore data to a consistent state at any time upstream of the rain ; At the same time, the ability to provide open data protocols , It can support other systems to subscribe to data changes ;

TiCDC characteristic

• High availability ： Can be multi node , Mutual interference , If a node fails , Automatic failover ;
• performance ：         commonly 3s Inside ;
• Support rich downstream formats ：      compatible MySQL Any of the formats DB;

TiCDC Architecture and principles

• TiDB receive SQL, TiKV After the nodes of are changed , Only the modification logs of this node are output to TiCDC Cluster colony
• TiCDC The node is responsible for receiving TiKV Log , And logically spliced together inside ;
• Multiple Capture Form a cluster , Some Capture Be responsible for pulling a part TiKV journal , And with others Capture synergy , Gather the received logs to a Capture Owner, And then send it to the downstream ;
• If one Capure fault , Will be owner Automatically transfer to other Capture, Then continue to assemble all the change logs , Output to downstream

TiCDC Applicable scenario

Suitable for data source （ Upstream database ） by TiCDC, Target database （ Downstream database ） For the class MySQL Database and Kafka; meanwhile TiCDC Open Protocol It is an aeronautical level data change notification protocol , Can be used for monitoring 、 cache 、 Full-text index 、 Analysis engine 、 The master-slave replication of heterogeneous databases provides data sources ; 

TiCDC Configuration requirements

Production environment ： 16c/64G/SSD/ Wan Zhao nic />2 node / >Centos 7.3

Deploy

install tidb If the cluster is not installed , You can use the scale-out Installation by ;

vim scale-out.yaml

cdc_servers:
  - host: XX.XX.XX.XX
    port: 8300
    deploy_dir: "/export/servers/TiDB/ticdc-8300/"
    log_dir: "/export/servers/TiDB/ticdc-8300/log"
  - host: XX.XX.XX.XX
    port: 8300
    deploy_dir: "/export/servers/TiDB/ticdc-8300/"
    log_dir: "/export/servers/TiDB/ticdc-8300/log"

Capacity expansion command ： tiup cluster scale-out cluster-default scale-out.yaml 

cdc Management tool installation ：  tiup install cdc

Version view

TiCDC]# tiup cdc version
Starting component `cdc`: /root/.tiup/components/cdc/v5.3.0/cdc version
Release Version: v5.3.0
Git Commit Hash: 20626babf21fc381d4364646c40dd84598533d66
Git Branch: heads/refs/tags/v5.3.0
UTC Build Time: 2021-11-22 10:37:02
Go Version: go version go1.16.4 linux/amd64
Failpoint Build: false

Create tasks

tiup cdc cli changefeed create --pd=${pd_address}:2379 --sink-url="mysql://[email protected]@127.0.0.1:3306" -changefeed-id="cdc-task1"

• --pd : adopt PD Can be obtained TiKV Metadata information ;
• --sink-url ： Downstream database address connection string
• --changefeed-id: Synchronization task id, If you don't specify ,ticdc It will automatically generate a UUID;
• --start-ts: Appoint changefeed At the beginning TSO; If omitted, start from the execution time ;
• --target-ts: Designated target TSO; After designation , As of the target time, the synchronization stops ; If omitted, it will be synchronized all the time ;

  Other relevant parameters

Downstream is MySQL/TiDB
worker-count: Concurrency of downstream execution
max-txn-row:   perform sql Of batch size

Downstream is Kafka
protocol： Output to kafka Message protocol , Optional default、canal、 avro、 maxwell;
max-message-bytes: Each direction Kafka broker The maximum amount of data to send a message , Default 64MB

Task query

cdc cli changefeed list --pd=http://${pd_ip}:2379  

You can specify multiple PD node

cdc cli changefeed query -s --pd=http://${pd}:2379 --changefeed-id=${task-name}

Pause / recovery / Delete synchronization task

cdc cli changefeed pause/resume/remove  --pd=http://${pd}:2379 --changefeed-id=${task-name}
 

to update TiCDC Synchronize task configuration
Pause --> Modify the configuration --> Recovery task

cdc cli changefeed pause -C test-cf --pd=http://${pd}:2379
cdc cli changefeed update ...
cdc cli changefeed resume ..

.TiCDC Limit

• Valid index ： There must be a primary key , Or not empty / Unique index of non virtual generated column ;
• It is not supported to use alone RawKV Of TiKV colony ;
• Temporary does not support TiDB Created in SEQUENCE Of DDL Operation and SEQUENCE function
• I won't support it TiKV Hibernate Region