当前位置:网站首页>Detailed explanation of debezium architecture of debezium synchronization
Detailed explanation of debezium architecture of debezium synchronization
2022-07-07 11:55:00 【Auspicious flight】
The most common is , You pass Apache Kafka Connect Deploy Debezium .Kafka Connect Is a framework and runtime for implementation and operation :
Send the record to Kafka Source connector for , for example Debezium
Record from Kafka The subject propagates to the receiver connector of other systems
The following figure shows a view based on Debezium Change the architecture of the data capture pipeline :
As shown in the figure , Deployed for MySQL and PostgresSQL Of Debezium Connector to capture changes to these two types of databases . Every Debezium Connectors establish connections to their source databases :
MySQL The connector uses the client library to access
binlog
.PostgreSQL The connector reads from the logical copy stream .
Kafka Connect As Kafka Separate services other than agents run .
By default , Changes from a database table will be written to Kafka The theme . if necessary , You can configure Debezium Of Topic routing conversion To adjust the target topic name . for example , You can :
Route records to topics with names different from table names
Streaming change event records from multiple tables to a single topic
The change event is recorded in Apache Kafka In the after ,Kafka Connect Different connectors in the ecosystem can stream records to other systems and databases , for example Elasticsearch、 Data warehouse and analysis system , or Infinispan Equal cache . According to the selected receiver connector , You may need to configure Debezium Of New record status extraction transformation . this Kafka Connect SMT take after
Structure from Debezium The change event of is propagated to the receiver connector . This replaces the detailed change event logging of the default propagation .
Debezium The server
Deploy Debezium Another way to do this is to use Debezium The server .Debezium The server is a configurable 、 Ready to use applications , It streams change events from the source database to various messaging infrastructures .
The following figure shows using the Debezium The architecture of the server's change data capture pipeline :
Debezium The server is configured to use Debezium One of the source connectors captures changes from the source database . Change events can be serialized into different formats , Such as JSON or Apache Avro, It will then be sent to one of the various messaging infrastructures , Such as Amazon Kinesis、Google Cloud Pub/Sub or Apache Pulsar.
Embedded engine
However , Use Debezium Another method of connector is Embedded engine . under these circumstances ,Debezium Will not pass Kafka Connect function , But as embedded into your customization Java Libraries in applications . This is important for using change events in the application itself ( No need to deploy full Kafka and Kafka Connect colony ) Or stream changes to Amazon Kinesis And other alternative messaging agents are very useful . You can find the latter in the sample repository Example .
边栏推荐
- Talk about SOC startup (IX) adding a new board to uboot
- [shortest circuit] acwing 1127 Sweet butter (heap optimized dijsktra or SPFA)
- R语言使用quantile函数计算评分值的分位数(20%、40%、60%、80%)、使用逻辑操作符将对应的分位区间(quantile)编码为分类值生成新的字段、strsplit函数将学生的名和姓拆分
- 总结了200道经典的机器学习面试题(附参考答案)
- The road to success in R & D efficiency of 1000 person Internet companies
- Neural approvals to conversational AI (1)
- STM32F1与STM32CubeIDE编程实例-315M超再生无线遥控模块驱动
- 问下flinkcdc2.2.0的版本,支持并发,这个并发是指多并行度吗,现在发现,mysqlcdc全
- 相机标定(1): 单目相机标定及张正友标定基本原理
- [encapsulation of time format tool functions]
猜你喜欢
What is cloud computing?
人大金仓受邀参加《航天七〇六“我与航天电脑有约”全国合作伙伴大会》
【神经网络】卷积神经网络CNN【含Matlab源码 1932期】
SwiftUI 教程之如何在 2 秒内实现自动滚动功能
Tsinghua Yaoban programmers, online marriage was scolded?
核舟记(一):当“男妈妈”走进现实,生物科技革命能解放女性吗?
超标量处理器设计 姚永斌 第8章 指令发射 摘录
In my limited software testing experience, a full-time summary of automation testing experience
The running kubernetes cluster wants to adjust the network segment address of pod
Visual Studio 2019 (LocalDB)\MSSQLLocalDB SQL Server 2014 数据库版本为852无法打开,此服务器支持782版及更低版本
随机推荐
Enclosed please find. Net Maui's latest learning resources
[question] Compilation Principle
从工具升级为解决方案,有赞的新站位指向新价值
UP Meta—Web3.0世界创新型元宇宙金融协议
竟然有一半的人不知道 for 与 foreach 的区别???
[filter tracking] comparison between EKF and UKF based on MATLAB extended Kalman filter [including Matlab source code 1933]
聊聊SOC启动(七) uboot启动流程三
Camera calibration (1): basic principles of monocular camera calibration and Zhang Zhengyou calibration
Camera calibration (2): summary of monocular camera calibration
How much do you know about excel formula?
Onedns helps college industry network security
Rationaldmis2022 array workpiece measurement
【神经网络】卷积神经网络CNN【含Matlab源码 1932期】
Electron adding SQLite database
Superscalar processor design yaoyongbin Chapter 8 instruction emission excerpt
Blog moved to Zhihu
Electron adding SQLite database
相机标定(2): 单目相机标定总结
5V串口接3.3V单片机串口怎么搞?
Improve application security through nonce field of play integrity API