Monitoring of debezium synchronization debezium

Catalog

    monitor Debezium Index of connector

    Enable in the local installation JMX

Zookeeper JMX environment variable

Kafka JMX environment variable

Kafka Connect JMX environment variable

    stay Docker Enable JMX

Docker Medium Zookeeper JMX environment variable

Docker Medium Kafka JMX environment variable

Docker Medium Kafka Connect JMX environment variable

    Use Prometheus and Grafana

   Java Flight records

---

You can use Apache Zookeeper、Apache Kafka and Kafka Connect Provided JMX Indicators to monitor Debezium. Use these indicators , You must start Zookeeper、Kafka and Kafka Connect Enable them when serving . Enable JMX It involves setting the correct environment variables . The environment variables you must set depend on whether you are installing locally or Docker Running in the container Zookeeper、Kafka and Kafka Connect.

If you run multiple services on the same machine , Please make sure to use different for each service JMX port .

    monitor Debezium Index of connector

Except yes Kafka、Zookeeper and Kafka Connect Medium JMX In addition to the built-in support of indicators , Each connector provides additional metrics , You can use these indicators to monitor their activities .

• MySQL Connector indicators

• MongoDB Connector indicators

• PostgreSQL Connector indicators

• SQL Server Connector indicators

• Db2 Connector indicators

• Oracle Connector indicators

• Cassandra Connector indicators

    Enable in the local installation JMX

Use Zookeeper、Kafka and Kafka Connect, You can enable JMX.

Zookeeper JMX environment variable

Zookeeper Built in right JMX Support for . Run with a local installation Zookeeper when ,zkServer.sh The script will recognize the following environment variables ：

JMXPORT

Enable JMX And specify that it will be used for JMX Port number . This value is used to specify JVM Parameters -Dcom.sun.management.jmxremote.port=$JMXPORT.

JMXAUTH

JMX Whether the client must use password authentication when connecting . Must be true or false. The default value is false.  This value is used to specify JVM Parameters -Dcom.sun.management.jmxremote.authenticate=$JMXAUTH.

JMXSSL

JMX Whether the client uses SSL/TLS Connect . Must be true or false. The default value is false.  This value is used to specify JVM Parameters -Dcom.sun.management.jmxremote.ssl=$JMXSSL.

JMXLOG4J

Whether it should be disabled Log4J JMX MBean. Must be true（ Default ） or false.  The default value is true.  This value is used to specify JVM Parameters -Dzookeeper.jmx.log4j.disable=$JMXLOG4J.

Kafka JMX environment variable

Run with a local installation Kafka when ,kafka-server-start.sh The script will recognize the following environment variables ：

JMX_PORT

Enable JMX And specify that it will be used for JMX Port number . This value is used to specify JVM Parameters -Dcom.sun.management.jmxremote.port=$JMX_PORT.

KAFKA_JMX_OPTS

JMX Options , During startup, it is passed directly to JVM. The default option is ：

• -Dcom.sun.management.jmxremote

• -Dcom.sun.management.jmxremote.authenticate=false

• -Dcom.sun.management.jmxremote.ssl=false

Kafka Connect JMX environment variable

Run with a local installation Kafka when ,connect-distributed.sh The script will recognize the following environment variables ：

JMX_PORT

Enable JMX And specify that it will be used for JMX Port number . This value is used to specify JVM Parameters -Dcom.sun.management.jmxremote.port=$JMX_PORT.

KAFKA_JMX_OPTS

JMX Options , During startup, it is passed directly to JVM. The default option is ：

• -Dcom.sun.management.jmxremote

• -Dcom.sun.management.jmxremote.authenticate=false

• -Dcom.sun.management.jmxremote.ssl=false

    stay Docker Enable JMX

If you pass Debezium Container image operation Apache Zookeeper、Kafka and Kafka Connect, Enable JMX You need several additional environment variables that are usually not needed when running on the local computer . This is because JVM You need to JMX The client notifies its host name . therefore ,Debezium Of Zookeeper、Kafka and Kafka Connect Container mirroring uses multiple environment variables to enable and configure JMX. Most of the environment variables for all images are the same , But there are some subtle differences .

Docker Medium Zookeeper JMX environment variable

The debezium/zookeeper Image recognition is related to JMX Related environment variables ：

JMXPORT（ Essential ）

Will be used for JMX Port number . This value is used to specify the following JVM Parameters ：

• -Dcom.sun.management.jmxremote.port=$JMXPORT

• -Dcom.sun.management.jmxremote.rmi.port=$JMXPORT

JMXHOST（ Essential ）

Docker The host IP Address or resolvable host name ,JMX Use it to construct send to JMX Client's URL.localhostor Value 127.0.0.1 It doesn't work . Usually ,0.0.0.0 have access to . This value is used to specify JVM Parameters -Djava.rmi.server.hostname=$JMXHOST.

JMXAUTH

JMX Whether the client must use password authentication when connecting . Must be true or false. The default value is false.  This value is used to specify JVM Parameters -Dcom.sun.management.jmxremote.authenticate=$JMXAUTH.

JMXSSL

JMX Whether the client uses SSL/TLS Connect . Must be true or false. The default value is false.  This value is used to specify JVM Parameters -Dcom.sun.management.jmxremote.ssl=$JMXSSL.

JMXLOG4J

Whether it should be disabled Log4J JMX MBean. Must be true or false. The default value is true.  This value is used to specify JVM Parameters -Dzookeeper.jmx.log4j.disable=$JMXLOG4J.

Example 1. stay Zookeeper Docker Enable... In container JMX

Following example Docker The command uses and environment variables debezium/zookeeper Value image boot container , And will Docker Host port 9010 Mapped to container JMX port ：JMXPORTJMXHOST

$ docker run -it --rm --name zookeeper -p 2181:2181 -p 2888:2888 -p 3888:3888 -p 9010:9010 -e JMXPORT=9010 -e JMXHOST=10.0.1.10 debezium/zookeeper:latest

Docker Medium Kafka JMX environment variable

The debezium/kafka Image recognition is related to JMX Related environment variables ：

JMXPORT（ Essential ）

Will be used for JMX Port number . This value is used to specify the following JVM Parameters ：

• -Dcom.sun.management.jmxremote.port=$JMXPORT

• -Dcom.sun.management.jmxremote.rmi.port=$JMXPORT

JMXHOST（ Essential ）

Docker The host IP Address or resolvable host name ,JMX Use it to construct send to JMX Client's URL.localhostor Value 127.0.0.1 It doesn't work . Usually ,0.0.0.0 have access to . This value is used to specify JVM Parameters -Djava.rmi.server.hostname=$JMXHOST.

JMXAUTH

JMX Whether the client must use password authentication when connecting . Must be true or false. The default value is false.  This value is used to specify JVM Parameters -Dcom.sun.management.jmxremote.authenticate=$JMXAUTH.

JMXSSL

JMX Whether the client uses SSL/TLS Connect . Must be true or false. The default value is false.  This value is used to specify JVM Parameters -Dcom.sun.management.jmxremote.ssl=$JMXSSL.

Example 2. stay Kafka Docker Enable... In container JMX

Following example Docker The command uses and environment variables debezium/kafka Value image boot container , And will Docker Host port 9011 Mapped to container JMX port ：JMXPORTHOST_NAME

$ docker run -it --rm --name kafka -p 9092:9092 -p 9011:9011 -e JMXPORT=9011 -e JMXHOST=10.0.1.10 --link zookeeper:zookeeper debezium/kafka:latest

Docker Medium Kafka Connect JMX environment variable

The debezium/connect Image recognition is related to JMX Related environment variables ：

JMXPORT（ Essential ）

Will be used for JMX Port number . This value is used to specify the following JVM Parameters ：

• -Dcom.sun.management.jmxremote.port=$JMXPORT

• -Dcom.sun.management.jmxremote.rmi.port=$JMXPORT

JMXHOST（ Essential ）

Docker The host IP Address or resolvable host name ,JMX Use it to construct send to JMX Client's URL.localhostor Value 127.0.0.1 It doesn't work . Usually ,0.0.0.0 have access to . This value is used to specify JVM Parameters -Djava.rmi.server.hostname=$JMXHOST.

JMXAUTH

JMX Whether the client must use password authentication when connecting . Must be true or false. The default value is false.  This value is used to specify JVM Parameters -Dcom.sun.management.jmxremote.authenticate=$JMXAUTH.

JMXSSL

JMX Whether the client uses SSL/TLS Connect . Must be true or false. The default value is false.  This value is used to specify JVM Parameters -Dcom.sun.management.jmxremote.ssl=$JMXSSL.

Example 3. stay Kafka Connect Docker Enable... In container JMX

Following example Docker The command uses and environment variables debezium/connect Value image boot container , And will Docker Host port 9012 Mapped to container JMX port ：JMXPORTJMXHOST

$ docker run -it --rm --name connect \
  -p 8083:8083 -p 9012:9012 \
  -e JMXPORT=9012 -e JMXHOST=10.0.1.10 \
  -e GROUP_ID=1 \
  -e CONFIG_STORAGE_TOPIC=my_connect_configs \
  -e OFFSET_STORAGE_TOPIC=my_connect_offsets \
  -e STATUS_STORAGE_TOPIC=my_connect_statuses \
  --link zookeeper:zookeeper \
  --link kafka:kafka \
  --link mysql:mysql \
  debezium/connect:latest

    Use Prometheus and Grafana

Debezium and Kafka Public indicators can be used Prometheus and Grafana Export and display . You can go to Debezium Sample Repository Find examples of the required configurations and sample dashboards for different connectors .

These dashboards are not Debezium Part of itself , Instead, try your best to maintain .

   Java Flight records

Kafka Connect Of Debezium The container image is also JDK Flight Recorder Provides easy-to-use support . To enable recording , Environment variables must be set ENABLE_JFR=true. Then recording will start automatically when the container starts . By default , Enable recording , That is, use the default settings for continuous recording , Then you can use jcmd or JDK Mission Control dump .

You can modify the behavior of the recorder . So , You need to set any environment variable prefixed with JFR_RECORDING_ To set recording parameters . The part after the prefix is converted to lowercase , Underline is replaced by dash , therefore JFR_RECORDING_PATH_TO_GC_ROOTS=true Turn into path-to-gc-roots=true.

Flight recorder control options can be through JFR_OPT_ Prefix environment variables .