Ogg synchronize MySQL data to greenplus

This article has participated in 「 New people's creation ceremony 」 Activities , Start the road of nuggets creation together .

Source end ：

ogg Installation , You can download and decompress them directly on the official website to the corresponding directory , Address ：www.oracle.com/technetwork… Version is 12.3.0.1.5

mysql Configure environment variables ：

export GG_HOME=/home/ogg/

export PATH= $PATH:$ GG_HOME export LD_LIBRARY_PATH= $OGG_HOME:$ JAVA_HOME/jre/lib/amd64/libjsig.so: $JAVA_HOME/jre/lib/amd64/server/libjvm.so:$ JAVA_HOME/jre/lib/amd64/server:$JAVA_HOME/jre/lib/amd64

mysql Turn on bin-log stay my.conf Add... To the configuration file

[mysqld]

server_id=1 log-bin=/home/mysql3307/mysql-5.6.33/logs/log-bin binlog_format=row

restart mysql

Get into ogg The installation directory , Execute the following command ： Get into ogg

ggsci

Create subdirectories

create subdirs

Edit extraction process

edit param ext_cshy

Here is the content ：

EXTRACT ext_cshy

setenv (MYSQL_HOME="/home/mysql3307/mysql-5.6.33") setenv (NLS_LANG="AMERICAN_AMERICA.UTF8") tranlogoptions altlogdest /home/mysql3307/mysql-5.6.33/data/slave3307_bin.index sourcedb [email protected]:xxxx USERID xxxx, PASSWORD xxxx DISCARDROLLOVER ON FRIDAY DISCARDFILE /home/ogg/dirrpt/ext_cshy.dsc, APPEND, MEGABYTES 1024 REPORTCOUNT EVERY 1 MINUTES, RATE GETUPDATEBEFORES UPDATERECORDFORMAT COMPACT GETTRUNCATES EXTTRAIL /home/ogg/dirdat/cs table aisino.wechat_mp_order; table aisino.wechat_mp_order_item;

Add extraction process

add extract ex_test,tranlog,begin now

add exttrail /home/ogg/dirdat/te,extract ex_test

Edit posting process

edit param pum_cshy

Here is the content ：

EXTRACT pum_cshy

SETENV (NLS_LANG = "AMERICAN_AMERICA.UTF8") PASSTHRU RMTHOST 192.168.x.xx, MGRPORT 7809 RMTTRAIL /data/ogg/dirdat/cs table aisino.wechat_mp_order; table aisino.wechat_mp_order_item add extract dp_test,exttrailsource /opt/ogg/dirdat/te

Add delivery process ( The path is the directory path posted to the remote )

add rmttrail /opt/ogg/dirdat/te,extract dp_test

Edit the mapping file

edit param def_cshy

Here is the content ：

defsfile /home/ogg/dirdef/def_cshy.prm

sourcedb [email protected]:xxxx, userid xxxx,password xxxx table aisino.wechat_mp_order; table aisino.wechat_mp_order_item;

sign out ogg

exit

Generate the corresponding description file by mapping the configuration file

./defgen paramfile ./dirprm/def_cshy.prm

Copy ./dirdef Under the defgen.prm To the target end ${OGG_HOME}/dirdef Next

Target end ：

install ogg Version is 12.3.0.1.2

Edit replication process

edit rep_test

Here is the content ：

REPLICAT rep_cshy

TARGETDB LIBFILE libggjava.so SET property=/data/ogg/dirprm/kafka_cshy.props SOURCEDEFS /data/ogg/dirdef/def_cshy.prm REPORTCOUNT EVERY 1 MINUTES, RATE GROUPTRANSOPS 10000 GETTRUNCATES MAP aisino., TARGET cshy.;

Add replication process

add replicat rep_test,exttrail /opt/ogg/dirdat/te,checkpointtable ogg.checkpoint

To configure kafka The configuration file dirprm/kafka_cshy.props Here is the content ：

gg.handlerlist = kafkahandler

gg.handler.kafkahandler.type=kafka gg.handler.kafkahandler.KafkaProducerConfigFile=custom_kafka_producer.properties #The following resolves the topic name using the short table name gg.handler.kafkahandler.topicMappingTemplate=ogg_cshy #The following selects the message key using the concatenated primary keys gg.handler.kafkahandler.format.includePrimaryKeys=true gg.handler.kafkahandler.format=json gg.handler.kafkahandler.BlockingSend =false gg.handler.kafkahandler.includeTokens=false gg.handler.kafkahandler.mode=op goldengate.userexit.writers=javawriter javawriter.stats.display=TRUE javawriter.stats.full=TRUE gg.log=log4j gg.log.level=INFO gg.report.time=30sec #Sample gg.classpath for Apache Kafka gg.classpath=/data/ogg/dirprm/:/home/gpadmin/software/kafka_2.12-2.1.0/libs/:/data/ogg/:/data/ogg/lib/ #Sample gg.classpath for HDP #gg.classpath=/etc/kafka/conf:/usr/hdp/current/kafka-broker/libs/* javawriter.bootoptions=-Xmx512m -Xms32m -Djava.class.path=ggjava/ggjava.jar

start-up Source end ：