Oracle database CDC to Kafka on docker

  1. Test environment preparation
docker pull lensesio/fast-data-dev
docker run -d --name kafka-lenses -p 2181:2181 -p 3030:3030 -p 8081:8081 -p 8082:8082 -p 8083:8083 -p 9092:9092 -v /home/kafka-lenses:/data -e ADV_HOST=192.168.60.165 lensesio/fast-data-dev
docker exec -it kafka-lenses /bin/bash
kafka-topics --zookeeper 127.0.0.1:2181 --create --topic oracle_test --partitions 3 --replication-factor 1
docker pull helowin/oracle_11g
docker run -d -p 1521:1521 --name oracle helowin/oracle_11g
docker exec -it oracle bash
su root
password:helowin
vi etc/profile
export ORACLE_HOME=/home/oracle/app/oracle/product/11.2.0/dbhome_2    
export ORACLE_SID=helowin
export PATH=$ORACLE_HOME/bin:$PATH
source /etc/profile
ln -s $ORACLE_HOME/bin/sqlplus /usr/bin
su - oracle
sqlplus /nolog
conn /as sysdba
SQL>shutdown immediate
SQL>startup mount
SQL>alter database archivelog;
SQL>alter database open;
SQL>alter database add supplemental log data (all) columns;
SQL>create role logmnr_role;
SQL>grant create session to logmnr_role;
SQL>grant execute_catalog_role,select any transaction ,select any dictionary to logmnr_role;
SQL>create user kafka identified by kafka;
SQL>grant logmnr_role to kafka;
SQL>grant connect,resource,dba to kafka;
SQL>alter user kafka quota unlimited on users;
docker cp kafka-connect-oracle-1.0.jar kafka:/opt/kafka/libs/kafka-connect-oracle-1.0.jar
docker cp ojdbc7.jar kafka:/opt/kafka/libs/ojdbc7.jar
docker exec -it kafka /bin/bash
bootstrap.servers=192.168.60.165:9092
rest.port=8083
rest.advertised.port=8083
connect-distributed.sh /opt/kafka/config/connect-distributed.properties
connector.class=com.ecer.kafka.connect.oracle.OracleSourceConnector
reset.offset=false
db.hostname=172.21.0.11
tasks.max=1
db.user.password=kafka
table.blacklist=KAFKA.AA
table.whitelist=KAFKA.MYKAFKA
db.user=kafka
db.port=1521
db.fetch.size=1
multitenant=false
topic=oracle_test
parse.dml.data=true
db.name=helowin
db.name.alias=test
docker exec kafka kafka-console-consumer.sh --bootstrap-server 172.21.0.11:9092 --from-beginning --topic oracle_test

--

--

--

spark kafak flink develop

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Private and public project problems: a guide on how to avoid them

Web Scraping — Part 2

Data Protection Rule ( DPR ) in Watson Knowledge Catalog

Outside of a bank vault with locked door

The Case(s) Of Postgres Not Using Index

Launching your First EC2 instance-

Why I’m Learning PHP in 2019!

My First Month Learning To Code

Java Virtual Machine(JVM)

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
hivefans

hivefans

spark kafak flink develop

More from Medium

Outbox Pattern using Change Data Capture (CDC): Kafka,Outbox,Debezium,Postgres,Springboot

Kafka Connector with Custom Transformation

JMX Exporter for Kafka Metrics