Change Data Capture - activoricordi/nijinsky GitHub Wiki

Debezium

Debezium is a distributed platform for Change Data Capture (CDC). Change Data Capture (CDC) is a technique used to track row-level changes in database tables in response to create, update and delete operations. Debezium tap into row-level changes in database tables and convert them into event streams. These event streams are then sent to a Event Broker. For instance, if we want to setup a change data capture system using PostgreSQL and Debezium, it will use the Debezium PostgrSQL connector. This connector stream database modifications from PostgreSQL to Event Broker topics (Kafka API). These topics could be found at Apache Kafka or Azure Event Hubs.

debezium example mysql

MySQL is an open-source relational database system. It still remains one of the most used structured database. The Change Data Capture can be done using the following methods:

The latest available version is debezium/example-mysql:1.5

# Use  debezium/example-mysql:${DEBEZIUM_VERSION} or debezium/example-mysql:1.5
mysql:
  image: debezium/example-mysql:1.5
  ports:
   - 3306:3306
  environment:
   - MYSQL_ROOT_PASSWORD=debezium
   - MYSQL_USER=mysqluser
   - MYSQL_PASSWORD=mysqlpw

This solution is described at the following Debezium Examples mySQL.

Debezium User Interface

## Development infrastructure for running and testing the Debezium UI
version: '3'
services:
  dbzui-zookeeper:
    container_name: dbzui-zookeeper
    image: debezium/zookeeper:1.5
    networks:
      - dbzui-network

  dbzui-kafka:
    container_name: dbzui-kafka
    image: debezium/kafka:1.5
    ports:
      - "9092:9092"
    depends_on:
      - dbzui-zookeeper
    environment:
      - ZOOKEEPER_CONNECT=dbzui-zookeeper:2181
    networks:
      - dbzui-network

  dbzui-db-pg:
    container_name: dbzui-db-pg
    image: debezium/example-postgres:1.5
    ports:
      - "65432:5432"
    environment:
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=postgres
    networks:
      - dbzui-network

  dbzui-db-mysql:
    container_name: dbzui-db-mysql
    image: debezium/example-mysql:1.5
    ports:
      - "63306:3306"
    environment:
      - MYSQL_ROOT_PASSWORD=debezium
      - MYSQL_USER=mysqluser
      - MYSQL_PASSWORD=mysql
    networks:
      - dbzui-network

  dbzui-db-mongo:
    container_name: dbzui-db-mongo
    image: mongo:3.6
    command: ["mongod", "--replSet", "rs0", "--auth"]
    ports:
      - "37017:27017"
    volumes:
    - ./backend/src/test/resources/initialize-mongo-single.js:/docker-entrypoint-initdb.d/initialize-mongo-single.js
    environment:
      - MONGO_INITDB_ROOT_USERNAME=admin
      - MONGO_INITDB_ROOT_PASSWORD=admin
      - MONGO_INITDB_DATABASE=admin
    networks:
      - dbzui-network

  mongo-initializer:
    image: mongo:3.6
    networks:
      - dbzui-network
    depends_on:
      - dbzui-db-mongo
    command: bash -c 'sleep 20 && mongo -u admin -p admin --authenticationDatabase admin dbzui-db-mongo:27017/inventory --eval "rs.initiate();"'

  dbzui-connect:
    container_name: dbzui-connect
    image: debezium/connect:1.5
    ports:
      - "8083:8083"
    depends_on:
      - dbzui-kafka
      - dbzui-db-pg
    environment:
      - BOOTSTRAP_SERVERS=dbzui-kafka:9092
      - GROUP_ID=1
      - CONFIG_STORAGE_TOPIC=my_connect_configs
      - OFFSET_STORAGE_TOPIC=my_connect_offsets
      - STATUS_STORAGE_TOPIC=my_connect_statuses
    networks:
      - dbzui-network

  dbzui-backend:
    container_name: dbzui-backend
    build:
      context: .
    ports:
      - "8080:8080"
    environment:
      KAFKA_CONNECT_URI: http://dbzui-connect:8083
    depends_on:
      - dbzui-connect
    networks:
      - dbzui-network
networks:
  dbzui-network:
    external: false