maxwell concept

maxwell definition

maxwell is a real-time capture MYSQL software written in java. It reads the MYSQL binary log Binlog in real time and generates JSON format, which is used as a producer to send to Kafka, Redis and other applications.

The Principle of Maxwell

Disguise yourself as a MYSQL salve, and then copy data from the master as a salve.

Mysql master-slave replication process

The master main library change record is written to the binary log file.
salve sends the dump protocol from the library I/O thread to the mysql master, and copies the binary log events of the master master library to the relay log of the slave library.
salve has a sql thread that will read the relay log,The sql events in this log are executed sequentially to be consistent with the main library.

binary log classification

binary log has three formats: statement, row, mixed

statement

At the statement level, binlog records the statement of the write operation each time, which saves space, but may produce different results, such as update test set create_date=now(). If you use binlog to reply to the log, the execution time will also be different., the resulting data is different.

row

Row level, binlog records the changes of each row of records after each operation, advantages: maintain data consistency, because no matter what operation is performed, it only records the effect after execution.Disadvantage: takes up a lot of space

mixed

Mixed level, the upgraded version of statement, the default is statement, but when the function contains UUID(); when the table containing the AUTO_INCREMENT field is updated; when the INSERT DELAYED statement is executed; when using UDF; it will be processed in the way of ROW.Advantages: It saves space and also takes into account a certain consistency.Disadvantages: There are also individual cases that can cause inconsistencies.

binlog format chosen by maxwell

Maxwell needs to monitor and analyze, and it is more suitable to choose the row format, which has strong consistency.

maxwell and canal comparison

Language: All written in java
Data format: canal format is free, maxwell can only send json messages
Data collection mode: canal only supports incremental collection, maxwell supports full and incremental
Data landing: Canal needs to be customized, maxwell supports kafka and other platforms
ha: all support