Maxwell is an easy-to-use software for capturing MySQL data in real time

Chapter one Maxwell summary

1.1、Maxwell brief introduction

Maxwell It's the United States Zendesk Open source , use Java Compiling MySQL Real time capture software .

Real time reading MySQL Binary log Binlog, And generate JSON Formatted message , Send to as producer Kafka,Kinesis、RabbitMQ、Redis、Google Cloud Pub/Sub、 Files or other platform applications .

Official website address ：http://maxwells-daemon.io/

Official website page ：

1.2、maxwell working principle

1.2.1、mysql Master slave replication process

1、Master The master library will change records , Write to binary log (binary log) in

2、Slave From library to mysql master send out dump agreement , take master The main library binary log events Copy to its trunk log (relay log);

3、Slave Read and redo the events in the relay log from the library , Synchronize the changed data to your own database .

1.2.2、Mysql Medium binlog

【1】 What is? binlog

MySQL The binary log of can be said MySQL The most important log , It records everything DDL and DML( In addition to data query statements ) sentence , Record as an event , It also contains the time consumed by statement execution ,MySQL The binary log of is transaction safe

Generally speaking, there will be 1% Loss of performance . Binary has two most important usage scenarios :

firstly ：MySQL Replication stay Master End open binlog,Master Pass its binary log to slaves In order to achieve master-slave Data consistent purpose .

second ： Naturally, it's data recovery , By using mysqlbinlog Tools to recover data .

Binary logs include two types of files ：

Binary log index file （ The file name suffix is .index） Used to record all Binary file

Binary log file （ The file name suffix is .00000*） Record everything in the database DDL and DML( In addition to data query statements ) Statement event .

【2】binlog On

First we need to find mysql The location of the configuration file for

/etc/my.cnf

stay mysql Under the configuration file of , Modify the configuration

stay [mysqld] block , Set up / add to log-bin=mysql-bin

The said binlog The prefix of the log is mysql-bin, The log files generated in the future are mysql-bin.000001 The numbers after the file are generated in order , Every time mysql Restart or reach the threshold of single file size , A new file , Number... In sequence .

【3】binlog Classification settings

mysql binlog There are three formats for , Namely STATEMENT,MIXED,ROW.

The difference between the three formats ：

1、statement

Sentence level ,binlog The statement that performs a write operation each time is recorded . relative row Mode saves space , But there can be inconsistencies , such as update test set create_date=now(); If you use binlog Log recovery , Due to the different execution time, the data may be different .

advantage ： Save a space

shortcoming ： It may cause data inconsistency .

2、row

Row level , binlog It records the changes of each line after each operation .

advantage ： Keep the data absolutely consistent . Because I don 't care sql What is it? , What functions are referenced , He only records the effect of execution .

shortcoming ： Take up a lot of space .

3、mixed

Blend level ,statement Upgraded version , To some extent, it solved statement The data inconsistency caused by some situations of the pattern

Acquiescence or statement, In some cases , for example ： When the function contains UUID() when ; contain AUTO_INCREMENT When the table of fields is updated ; perform INSERT DELAYED When the sentence is ; use UDF when ; According to ROW The way to deal with

advantage ： Save a space , At the same time, a certain degree of consistency is taken into account .

shortcoming ： There are also very few cases that still cause inconsistencies , in addition statement and mixed For the need to be, for binlog Monitoring is inconvenient .

Compare the above ,Maxwell Want to do monitoring analysis , choice row The format is more appropriate

1.2.3、maxwell working principle

Maxwell It works very simply , Namely Disguise yourself as MySQL One of the slave, And then to slave Pretend to be from MySQL(master) Copy the data

1.2.4、Maxwell and Canal The difference between

Chapter two Maxwell Installation

2.1、 Download address

（1）Maxwell Official website address ：http://maxwells-daemon.io/

（2） Document viewing address ：http://maxwells-daemon.io/quickstart/

But when we look at the version update log, we will find ：

Because my machine is installed with jdk1,8, So here we use 1.30.0 Previous version

2.2、Maxwell Installation and deployment

Environmental preparation ：

Make sure that... Is installed on the machine kafka and mysql,zookeeper

1、 take maxwell-1.29.2.tar.gz To /opt/software Next

2、 take maxwell Unzip to /opt/module that will do

3、 see maxwell Directory structure of

2.3、mysql Environmental preparation

1、 modify mysql Configuration file for , Turn on mysql Of binlog Set up

sudo vim /etc/my.cnf


server_id=1
#  Set the prefix of the generated binary file 
log-bin=mysql-bin
#  Set up binlog Log level of binary file of   Row level mode 
binlog_format=row
# binlog The execution Library of   If you do not add this parameter, then mysql Corresponding... Will be generated for all libraries binlog  That is to say, we should devote ourselves to all the libraries binlog monitor 
#  Set to monitor only one or some databases 
binlog-do-db=test_maxwell
binlog-do-db=test_maxwell1

2、 restart mysql service

sudo systemctl restart mysqld

3、 Sign in mysql And check whether the modification is successful

Log in mysql Type the following statement

show variables like '%binlog%';

4、 Get into /var/lib/mysql Catalog , see MySQL Generated binlog file

notes ：MySQL Generated binlog The initial size of the file must be 154 byte , Then the prefix is log-bin Parameter matching

Placed , The suffix is defaulted from .000001, Then increase in sequence .

except binlog Outside the file ,MySQL An additional one will be produced .index The index file is used to record the currently used binlog file .

5、 So let's test that out

【1】 Use sqlyog Connect to mysql

Look at the binary file again

2.4、 initialization Maxwell Meta database

1、 stay MySQL Create a maxwell Library for storage Maxwell Metadata

CREATE DATABASE maxwell;

When we use it, it will create corresponding tables by itself , Here we do not need to create our own tables . You don't know which tables to create

2、 Set up mysql User password security level

set global validate_password_length=4;
set global validate_password_policy=0;

3、 Assign an account to operate the database

GRANT ALL ON maxwell.* TO 'maxwell'@'%' IDENTIFIED BY '123456';

4、 Assign this account to monitor the permissions of other databases

GRANT SELECT ,REPLICATION SLAVE , REPLICATION CLIENT ON *.* TO [email protected]'%';

5、 Refresh mysql Table permissions

flush privileges;

The third chapter Maxwell Use

3.1、maxwell The start of the process

maxwell There are two ways to start a process

1、 Start... With command line arguments Maxwell process

bin/maxwell --user='maxwell' --password='123456' --host='hadoop02' --producer=stdout

Parameter interpretation ：

--user Connect mysql Users of

--password Connect mysql The password of the user

--host mysql Host name of the installation

--producer Producer mode (stdout： Console kafka：kafka colony )

test ：

2、 Profile startup maxwell

【1】 A copy of config.properties.example file

【2】 modify config.properties file

【3】 start-up

bin/maxwell --config ./config.properties

3.2、maxwell Case practice

3.2.1、 monitor Mysql Data and print on the console

【1】 function maxwell monitor mysql Data update

bin/maxwell --user='maxwell' --password='123456' --host='hadoop02' --producer=stdout

【2】 towards mysql Of test_maxwell Library user Table inserts a piece of data , see maxwell The console output

Output json Format data

{
  "database": "test_maxwell",
  "table": "user",
  "type": "insert",
  "ts": 1653211725,
  "xid": 2319,
  "commit": true,
  "data": {
    "id": 2,
    "name": " Li Si "
  }
}

Second timestamps

【3】 towards mysql Of test_maxwell Library user Table insert at the same time 3 Data , The console appears 3 strip json journal , explain maxwell It collects logs in data behavior units .

INSERT INTO USER VALUES (3," zhang wuji "),(4," The Monkey King "),(3," Pig eight quit ");

【4】 modify test_maxwell Library user A piece of data in the table , see maxwell The console output

{
  "database": "test_maxwell",
  "table": "user",
  "type": "update",
  "ts": 1653212058,
  "xid": 3061,
  "commit": true,
  "data": {
    "id": 2,
    "name": " Wang Wu "
  },
  "old": {
    "name": " Li Si "
  }
}

【5】 Delete a piece of data

{
  "database": "test_maxwell",
  "table": "user",
  "type": "delete",
  "ts": 1653212134,
  "xid": 3234,
  "commit": true,
  "data": {
    "id": 3,
    "name": " Pig eight quit "
  }
}

3.2.2、 monitor Mysql Data output to kafka

1、 start-up zookeeper Clusters and kafka colony

2、 start-up Maxwell monitor binlog

bin/maxwell --user='maxwell' --password='123456' --host='hadoop02' --producer=kafka --kafka.bootstrap.servers=hadoop02:9092 --kafka_topic=maxwell

Be careful We do not update the data here topic Will not be created

We used it kafka Graphic tools for kafka tool open

Let's go back to mysql Update data in

see topics

once mysql The table is updated with data , So the bottom binlog There must be a change ,binlog Change then we · Of maxwell The process can capture this · change , Once captured, this data will be transmitted to kafka Inside .

3、 open kafka Consumer consumption of the console maxwell The theme

kafka-console-consumer.sh --bootstrap-server hadoop02:9092 --topic maxwell

adopt kafka Consumers come to view the data , This indicates that the data was successfully transferred into kafka

Here we will modify another database , Because of the previous setup, we used binlog Monitored two databases , One is test_maxwell, One is test_maxwell1, Now we are test_maxwell1 Insert a piece of data into the table of

see kafka The theme of ：

So the next step is to introduce customization startup maxwell Process output to kafka

3.2.3、Kafka Topic partition control

In the actual production environment , We use them all the time maxwell Monitor multiple mysql Database data , Then send the data to kafka A theme of Topic, And this theme must be multi partitioned , To improve concurrency .

So how to control the partition of these data , It's becoming critical , The implementation steps are as follows ：

【1】 Manually create three partitions topic

【2】 modify maxwell Configuration file for , Customized start maxwell process

modify producer The model is kafka

Specify the subject to send

Set partition parameters

Save and exit ！

【3】 Start by using the configuration file maxwell process

bin/maxwell --config ./config.properties

【4】 towards test_maxwell Library test Insert a piece of data into the table again

【5】 towards test_maxwell1 Library test Insert a piece of data into the table again

Look again. topic

It indicates that data from different databases will be sent to different partitions

3.2.4、 monitor mysql Specifies that table data is output to the console

In the previous operations, we monitored all the tables under the database , So how to monitor the specified table ？

Use --filter Parameter to filter

First of all we have test_maxwell Create a new table in the database student

【1】 function maxwell To monitor mysql Specify table data update

bin/maxwell --user='maxwell' --password='123456' --host='hadoop02' --filter 'exclude: *.*, include:test_maxwell.student' --producer=stdout

This means that only student surface

to update user Table data ：

to update student Table data ：

You can also set include:test_maxwell.*, Monitor... In this way mysql All tables of a library , That is, filter the entire library

3.2.5、 monitor mysql Specify that the full table data is output to the console

Data initialization

Maxwell By default, the process can only monitor mysql Of binlog Log addition and change data , however Maxwell It supports data initialization , It can be modified by Maxwell Metadata , Come on MySQL Data initialization for a table of , That's what we often say Full amount of synchronization .

demand ： take test_maxwell Under the Treasury test2 The four pieces of data in the table , Import the whole amount into maxwell Print from the console .

【1】 modify Maxwell Metadata , Trigger data initialization mechanism , stay mysql Of maxwell In the library bootstrap Insert a piece of data into the table , Specify the database name and table name that need full data

insert into maxwell.bootstrap(database_name,table_name) values('test_maxwell','user');

【2】 start-up maxwell process , At this time, the initialization program will print directly test2 All data of table

bin/maxwell --user='maxwell' --password='123456' --host='hadoop02' --producer=stdout

【3】 When all the data is initialized ,Maxwell Your metadata will change

is_complete Fields from 0 Turn into 1

start_at Fields from null Into a specific time ( Data synchronization start time )

complete_at Fields from null Into a specific time ( Data synchronization end time )