Oracle database CDC to Kafka on docker

  1. Test environment preparation
docker pull lensesio/fast-data-dev
docker run -d --name kafka-lenses -p 2181:2181 -p 3030:3030 -p 8081:8081 -p 8082:8082 -p 8083:8083 -p 9092:9092 -v /home/kafka-lenses:/data -e ADV_HOST=192.168.60.165 lensesio/fast-data-dev
docker exec -it kafka-lenses /bin/bash
kafka-topics --zookeeper 127.0.0.1:2181 --create --topic oracle_test --partitions 3 --replication-factor 1
docker pull helowin/oracle_11g
docker run -d -p 1521:1521 --name oracle helowin/oracle_11g
docker exec -it oracle bash
su root
password:helowin
vi etc/profile
export ORACLE_HOME=/home/oracle/app/oracle/product/11.2.0/dbhome_2    
export ORACLE_SID=helowin
export PATH=$ORACLE_HOME/bin:$PATH
source /etc/profile
ln -s $ORACLE_HOME/bin/sqlplus /usr/bin
su - oracle
sqlplus /nolog
conn /as sysdba
SQL>shutdown immediate
SQL>startup mount
SQL>alter database archivelog;
SQL>alter database open;
SQL>alter database add supplemental log data (all) columns;
SQL>create role logmnr_role;
SQL>grant create session to logmnr_role;
SQL>grant execute_catalog_role,select any transaction ,select any dictionary to logmnr_role;
SQL>create user kafka identified by kafka;
SQL>grant logmnr_role to kafka;
SQL>grant connect,resource,dba to kafka;
SQL>alter user kafka quota unlimited on users;
docker cp kafka-connect-oracle-1.0.jar kafka:/opt/kafka/libs/kafka-connect-oracle-1.0.jar
docker cp ojdbc7.jar kafka:/opt/kafka/libs/ojdbc7.jar
docker exec -it kafka /bin/bash
bootstrap.servers=192.168.60.165:9092
rest.port=8083
rest.advertised.port=8083
connect-distributed.sh /opt/kafka/config/connect-distributed.properties
connector.class=com.ecer.kafka.connect.oracle.OracleSourceConnector
reset.offset=false
db.hostname=172.21.0.11
tasks.max=1
db.user.password=kafka
table.blacklist=KAFKA.AA
table.whitelist=KAFKA.MYKAFKA
db.user=kafka
db.port=1521
db.fetch.size=1
multitenant=false
topic=oracle_test
parse.dml.data=true
db.name=helowin
db.name.alias=test
docker exec kafka kafka-console-consumer.sh --bootstrap-server 172.21.0.11:9092 --from-beginning --topic oracle_test

--

--

--

spark kafak flink develop

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Help, I need to develop a detection use case.

Use case development and maintenance is a structured process. Especially when you want to establish a high-velocity development/maintenance team.

Which programming language should I choose as a beginner?

Introduction to Docker and Container-based development

Turning the Dials in Elasticsearch

Anthos vs ECS Anywhere, The Fight For Hybrid Cloud Orchestration

What does a developer look for in a new job (except for money)?

A visual guide to Google Maps API Key generation, billing and quota limits set-up

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
hivefans

hivefans

spark kafak flink develop

More from Medium

Event Streaming Platform

event streaming platform

Querying Cache in Apache Ignite

Kafka Cluster on Docker Compose

Integrating Kafka with Coherence