MaxWell scraped data

MaxWell installation and configuration

Prepare the MaxWell package in advance

 tar -zxvf /opt/software/maxwell-1.25.0.tar.gz -C /opt/module/

1. Create a maxwell library in MySQL to store Maxwell metadata

mysql -uroot -p000000mysql> CREATE DATABASE maxwell;

2. Set the security level

mysql> set global validate_password_length=4;mysql> set global validate_password_policy=0;

3. Assign an account to operate the database

GRANT ALL ON maxwell.* TO 'maxwell'@'%' IDENTIFIED BY '000000';

4. Assign this account to monitor the permissions of other databases

GRANT SELECT , REPLICATION SLAVE , REPLICATION CLIENT ON *.* TO [email protected]'%';

Maxwell monitoring captures MySQL data

1. Copy the configuration file

cp config.properties.example config.properties

2. Modify the configuration file

producer=kafkakafka.bootstrap.servers=hadoop102:9092,hadoop103:9092,hadoop104:9092#need to addkafka_topic=ods_base_db_m# mysql login infohost=hadoop102user=maxwellpassword=000000#Need to add initialization will be usedclient_id=maxwell_1

Note:

• The default is to output to a kafka partition of the specified Kafka topic, because multiple partitions in parallel may disrupt the order of binlog
• If you want to improve the degree of parallelism, first set the number of partitions of kafka>1, and then set the producer_partition_by property
  • producer_partition_by=database|table|primary_key|random| column

3. Start maxwell

/opt/module/maxwell-1.25.0/bin/maxwell --config /opt/module/maxwell-1.25.0/config.properties

4. Start the kafka consumer client (the premise is to start zookeeper first)

bin/kafka-console-consumer.sh --bootstrap-server hadoop102:9092 --topic ods_base_db_m

5. Operate the database to discover consumer consumption information