CDC MYSQL :KAFKA SOURCE JDBC Setup - mahuadasgupta/kafkasetup GitHub Wiki

Use Case:

Here ,the need of the hour is to do a POC around CDC concept. change data capture is a concept in which if a new record is inserted in a table then its taken as change in source . lets elaborate the same by below step.

1)  Connect to a rdbms (mysql) ( mysql -u root)
2)  Connect to a tablespace/databas(pocTest)( use pocTest)
3)  Create a small table schema  
CREATE TABLE Persons ( Id int NOT NULL AUTO_INCREMENT, LastName varchar(255) NOT NULL, FirstName varchar(255), ts TIMESTAMP DEFAULT CURRENT_TIMESTAMP NOT NULL ,PRIMARY KEY (Id) );
3) configure the source properties file with below content  /opt/confluent-4.1.0/etc/kafka-connect-jdbc/source-mysql-jdbc.properties

name=test-source-mysql-jdbc-autoincrement
connector.class=io.confluent.connect.jdbc.JdbcSourceConnector
tasks.max=1
connection.url=jdbc:mysql://127.0.0.1:3306/pocTest?user=root
table.whitelist=accounts,Persons
#query=select * from accounts
#mode=incrementing
mode=timestamp
timestamp.column.name=ts
incrementing.column.name=id
topic.prefix=test-
 
4) start schemregistry
./schema-registry-start -daemon ../etc/schema-registry/schema-registry.properties( if not allready  running)



6) start the connector in new putty 
cd  /opt/confluent-4.1.0/bin

export CLASSPATH=$CLASSPATH:.:/opt/confluent-4.1.0//share/java/kafka-connect-jdbc/mysql-connector-java.jar:/opt/confluent-4.1.0//share/java/kafka-connect-jdbc/kafka-connect-jdbc-4.1.0.jar

./connect-standalone ../etc/schema-registry/connect-avro-standalone.properties ../etc/kafka-connect-jdbc/source-mysql-jdbc.properties

7) go to the terminal where you created the table in (step 1,3)
insert some data on table
 insert into Persons(LastName,FirstName) values('dasgupta','mahua');  this all i will change later  
8)open  a putty terminal cd /opt/confluent-4.1.0/bin

./kafka-avro-console-consumer --bootstrap-server localhost:9092 --topic test-Persons --from-beginning
 it should start showing below.
{"Id":1,"LastName":"mogili","FirstName":{"string":pratyusha"},"ts":1525288606000}
9) repeat  7 couple of times , watch the automnatic update on the  step 8 

{"Id":1,"LastName":"das","FirstName":{"string":"mahua"},"ts":1525288606000}
{"Id":2,"LastName":"dasgupta","FirstName":{"string":"khounish"},"ts":1525289068000}


⚠️ **GitHub.com Fallback** ⚠️