Detailed explanation of debezium architecture of debezium synchronization

The most common is , You pass Apache Kafka Connect Deploy Debezium .Kafka Connect Is a framework and runtime for implementation and operation ：

• Send the record to Kafka Source connector for , for example Debezium

• Record from Kafka The subject propagates to the receiver connector of other systems

The following figure shows a view based on Debezium Change the architecture of the data capture pipeline ：

As shown in the figure , Deployed for MySQL and PostgresSQL Of Debezium Connector to capture changes to these two types of databases . Every Debezium Connectors establish connections to their source databases ：

• MySQL The connector uses the client library to access binlog.

• PostgreSQL The connector reads from the logical copy stream .

Kafka Connect As Kafka Separate services other than agents run .

By default , Changes from a database table will be written to Kafka The theme . if necessary , You can configure Debezium Of Topic routing conversion To adjust the target topic name . for example , You can ：

• Route records to topics with names different from table names

• Streaming change event records from multiple tables to a single topic

The change event is recorded in Apache Kafka In the after ,Kafka Connect Different connectors in the ecosystem can stream records to other systems and databases , for example Elasticsearch、 Data warehouse and analysis system , or Infinispan Equal cache . According to the selected receiver connector , You may need to configure Debezium Of New record status extraction transformation . this Kafka Connect SMT take after Structure from Debezium The change event of is propagated to the receiver connector . This replaces the detailed change event logging of the default propagation .

   Debezium The server

Deploy Debezium Another way to do this is to use Debezium The server .Debezium The server is a configurable 、 Ready to use applications , It streams change events from the source database to various messaging infrastructures .

The following figure shows using the Debezium The architecture of the server's change data capture pipeline ：

Debezium The server is configured to use Debezium One of the source connectors captures changes from the source database . Change events can be serialized into different formats , Such as JSON or Apache Avro, It will then be sent to one of the various messaging infrastructures , Such as Amazon Kinesis、Google Cloud Pub/Sub or Apache Pulsar.

    Embedded engine

However , Use Debezium Another method of connector is Embedded engine . under these circumstances ,Debezium Will not pass Kafka Connect function , But as embedded into your customization Java Libraries in applications . This is important for using change events in the application itself （ No need to deploy full Kafka and Kafka Connect colony ） Or stream changes to Amazon Kinesis And other alternative messaging agents are very useful . You can find the latter in the sample repository Example .