当前位置:网站首页>Detailed explanation of debezium architecture of debezium synchronization
Detailed explanation of debezium architecture of debezium synchronization
2022-07-07 11:55:00 【Auspicious flight】
The most common is , You pass Apache Kafka Connect Deploy Debezium .Kafka Connect Is a framework and runtime for implementation and operation :
Send the record to Kafka Source connector for , for example Debezium
Record from Kafka The subject propagates to the receiver connector of other systems
The following figure shows a view based on Debezium Change the architecture of the data capture pipeline :
As shown in the figure , Deployed for MySQL and PostgresSQL Of Debezium Connector to capture changes to these two types of databases . Every Debezium Connectors establish connections to their source databases :
MySQL The connector uses the client library to access
binlog
.PostgreSQL The connector reads from the logical copy stream .
Kafka Connect As Kafka Separate services other than agents run .
By default , Changes from a database table will be written to Kafka The theme . if necessary , You can configure Debezium Of Topic routing conversion To adjust the target topic name . for example , You can :
Route records to topics with names different from table names
Streaming change event records from multiple tables to a single topic
The change event is recorded in Apache Kafka In the after ,Kafka Connect Different connectors in the ecosystem can stream records to other systems and databases , for example Elasticsearch、 Data warehouse and analysis system , or Infinispan Equal cache . According to the selected receiver connector , You may need to configure Debezium Of New record status extraction transformation . this Kafka Connect SMT take after
Structure from Debezium The change event of is propagated to the receiver connector . This replaces the detailed change event logging of the default propagation .
Debezium The server
Deploy Debezium Another way to do this is to use Debezium The server .Debezium The server is a configurable 、 Ready to use applications , It streams change events from the source database to various messaging infrastructures .
The following figure shows using the Debezium The architecture of the server's change data capture pipeline :
Debezium The server is configured to use Debezium One of the source connectors captures changes from the source database . Change events can be serialized into different formats , Such as JSON or Apache Avro, It will then be sent to one of the various messaging infrastructures , Such as Amazon Kinesis、Google Cloud Pub/Sub or Apache Pulsar.
Embedded engine
However , Use Debezium Another method of connector is Embedded engine . under these circumstances ,Debezium Will not pass Kafka Connect function , But as embedded into your customization Java Libraries in applications . This is important for using change events in the application itself ( No need to deploy full Kafka and Kafka Connect colony ) Or stream changes to Amazon Kinesis And other alternative messaging agents are very useful . You can find the latter in the sample repository Example .
边栏推荐
- 软件内部的定时炸弹:0-Day Log4Shell只是冰山一角
- 竟然有一半的人不知道 for 与 foreach 的区别???
- STM32 entry development uses IIC hardware timing to read and write AT24C08 (EEPROM)
- Easyui学习整理笔记
- EasyUI learn to organize notes
- [filter tracking] strapdown inertial navigation pure inertial navigation solution matlab implementation
- 人大金仓受邀参加《航天七〇六“我与航天电脑有约”全国合作伙伴大会》
- What is high cohesion and low coupling?
- 本地navicat连接liunx下的oracle报权限不足
- Programming examples of stm32f1 and stm32subeide -315m super regenerative wireless remote control module drive
猜你喜欢
[filter tracking] strapdown inertial navigation simulation based on MATLAB [including Matlab source code 1935]
超标量处理器设计 姚永斌 第9章 指令执行 摘录
Enclosed please find. Net Maui's latest learning resources
相机标定(2): 单目相机标定总结
Flet教程之 14 ListTile 基础入门(教程含源码)
超标量处理器设计 姚永斌 第10章 指令提交 摘录
一度辍学的数学差生,获得今年菲尔兹奖
千人規模互聯網公司研發效能成功之路
清华姚班程序员,网上征婚被骂?
Fleet tutorial 19 introduction to verticaldivider separator component Foundation (tutorial includes source code)
随机推荐
Rationaldmis2022 array workpiece measurement
[filter tracking] strapdown inertial navigation simulation based on MATLAB [including Matlab source code 1935]
The annual salary of general test is 15W, and the annual salary of test and development is 30w+. What is the difference between the two?
MATLAB實現Huffman編碼譯碼含GUI界面
R language uses the quantile function to calculate the quantile of the score value (20%, 40%, 60%, 80%), uses the logical operator to encode the corresponding quantile interval (quantile) into the cla
一度辍学的数学差生,获得今年菲尔兹奖
【纹理特征提取】基于matlab局部二值模式LBP图像纹理特征提取【含Matlab源码 1931期】
SwiftUI 教程之如何在 2 秒内实现自动滚动功能
通过环境变量将 Pod 信息呈现给容器
18 basic introduction to divider separator component of fleet tutorial (tutorial includes source code)
【系统设计】指标监控和告警系统
powershell cs-UTF-16LE编码上线
【神经网络】卷积神经网络CNN【含Matlab源码 1932期】
STM32F1与STM32CubeIDE编程实例-315M超再生无线遥控模块驱动
Le Cluster kubernets en cours d'exécution veut ajuster l'adresse du segment réseau du pod
sql里,我想设置外键,为什么出现这个问题
Talk about SOC startup (11) kernel initialization
How to write test cases for test coupons?
Explore cloud database of cloud services together
千人規模互聯網公司研發效能成功之路