canal scrape data

canal installation and configuration

Prepare the canal package in advance

Note: canal is scattered after decompression, so it needs to be decompressed in a specific canal directory

module>mkdir canaltar -zxvf canal.deployer-1.1.4.tar.gz -C /opt/module/canal

1. Set the security level

mysql> set global validate_password_length=4;mysql> set global validate_password_policy=0

2. Assign an account (canal) to operate the database

mysql> GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'canal'@'%' IDENTIFIED BY 'canal' ;

canal monitoring captures MySQL data

1. Modify canal/conf/canal.properties

canal.serverMode = kafkacanal.mq.servers = hadoop102:9092,hadoop103:9092,hadoop104:9092

2. Modify conf/example/instance.properties

canal.instance.master.address=hadoop102:3306canal.instance.dbUsername=canalcanal.instance.dbPassword=canalcanal.mq.topic=ods_base_db_c

3. Start canal

canal>bin/startup.sh

4. Start the kafka consumer client test

Enable zookeeper and kafak in advance

bin/kafka-console-consumer.sh --bootstrap-server hadoop102:9092 --topic ods_base_db_c

5. Operate the database and view the effect of the kafak consumer client