Canal introduction

canal effect

canal It's an open source project of Alibaba , pure Java Development . Incremental log parsing based on Database , Provide incremental data subscription & consumption , At present, it mainly supports MySQL（ Also support mariaDB）. canal It's just one. A tool for synchronizing incremental data .

canal background

canal [kə'næl], Waterways / The Conduit / Ditch , The main use is based on MySQL Database incremental log parsing , Provide incremental data subscription and consumption

In the early days, Alibaba was deployed due to the dual computer rooms in Hangzhou and the United States , There is a business demand for synchronization across computer rooms , The implementation is mainly based on the business trigger Get incremental changes . from 2010 Year begins , The business gradually tries to get incremental changes from database log parsing for synchronization , Thus, a large number of incremental database subscription and consumption businesses are derived .

Incremental subscription and consumption based on logs include

• database mirroring
• Database real-time backup
• Index building and real-time maintenance ( Split heterogeneous indexes 、 Inverted index, etc )
• Business cache Refresh
• Incremental data processing with business logic Current canal Support source side MySQL The version includes 5.1.x , 5.5.x , 5.6.x , 5.7.x , 8.0.x

come from Alibaba official Introduce

canal Application scenarios

flow chart

1. Send the user's order information to the background .
2. The background server saves the order information to mysql database .
3. also canal monitor mysql Write operation changes in , Will be modified (Insert) Data written to kafka
4. adopt sparkStreaming Read Kafka Data in , Calculate .
5. The calculated result , Rewrite to the server , And return to the browser .

demand

According to the demand , Think about how to deal with ？

Get user's order information , Save stock in , And carry out real-time calculation （A Number of commodity transaction orders ,B Number of commodity transaction orders ...).

Why use canal?

Through the demand, we know that we need to calculate the order information , And real-time statistics of the transaction information of each commodity , At this time A goods 100 strip ,B goods 20 strip . This product information is dynamic , Every time a user submits , The background should calculate the latest commodity order records . So we need to accumulate in real time , For example, another batch of goods , among A goods 10 strip ,B goods 5 strip ,C goods 10 strip . Then the result of the display is A110 strip ,B 30 strip .C5 strip . But it is impossible to start again , in other words ,A+B+C =145 Medium data , Re divide A,B,C Information of each commodity . Instead, we should accumulate Like the last time A goods 100 And this time A Commodity 10 strip . Such efficiency is higher .

canal The role of

After understanding the above reasons , Let's talk about canal The role of , It can realize incremental synchronization , Or take A Commodity examples , In the first batch of data ,A The goods are 100 strip ,canal This new batch of data will be written into Kafka, Give it back spark Handle . Another batch of data is added for the second time , therefore canal Write the monitored new data to Kafka in . By analogy , In the end by the spark The calculated result is returned .

canal Except for writing kafka It can also write data to other middleware （mysql、elasticsearch、hbase etc. ）

Working diagram

---

working principle website

MySQL Principle of active / standby replication

image.png

• MySQL master Write data changes to binary log ( binary log, The records are called binary log events binary log events, Can pass show binlog events To view the )
• MySQL slave take master Of binary log events Copy to its trunk log (relay log)
• MySQL slave replay relay log Middle event , Change the data to reflect its own data

canal working principle

• canal simulation MySQL slave Interaction protocol , Pretend to be MySQL slave , towards MySQL master send out dump agreement
• MySQL master received dump request , Start pushing binary log to slave ( namely canal )
• canal analysis binary log object ( Originally byte flow )

---

BinaryLog

WAL：

• hbase When writing , First record the command of write operation in WAL In the log , Then write the data to memstore, This is to regionSerever After the process exits abnormally and restarts , Restore data , This mechanism is called WAL. WAL journal ： Call it a backup of write commands
• NameNode During operation , The metadata generated by the client in real time will be recorded in edits In file , stay namenode When restarting , Put the previous fsimage Document and edits Merge to get the latest element data . edtis： Client write operation named backup
• mysql Can be opened biglog Logging function , Every time after opening mysql Server send Write operations command , Will be recorded in a special file , This special file nine is called biglog journal . biglog journal ： Client write operation named backup , Its purpose is also for the abnormal exit of the server , Restore data ！

RelayLog

Slave (slave) How to synchronize the host (master) What about data ？ He will start two threads (I/O thread、SQL thread),

• I/O thread： Copy read host (master) Write to binary log What's new . And write the content to relay logs (relay log) in , Used for temporary buffer .
• SQL thread： For reading relay logs (relay log) Medium data of , And implement , such Slave (slave) There is and host (master) The same operation .

---

canal working principle （ My personal understanding ）

canal Like a " The spy ", It disguises itself as a Slave (slave), from host (master) Swindle data . After getting the data , And then analyze the data , such as · Just add new data or delete or modify data , And repackage the data , Rewrite the data into the third-party service （mysql、Kafka、es etc. ）.