stream replication debezium server - ghdrako/doc_snipets GitHub Wiki

Install

  1. Download
wget https://repo1.maven.org/maven2/io/debezium/debezium-server-dist/3.0.4.Final/debezium-server-dist-3.0.4.Final.tar.gz
tar -xzf debezium-server-dist-3.0.4.Final.tar.gz
cd debezium-server
  1. Configure W katalogu debezium-server/config wymagany jest pliku konfiguracyjnego application.properties lub application.yaml

Konfiguracja polaczenia do postgres z ssl-em:

Wyjaśnienie kluczy SSL:

  • debezium.database.sslmode - Ustawia tryb SSL. Możliwe wartości to:
    • disable: Brak SSL.
    • require: Wymaga SSL, ale nie weryfikuje certyfikatu.
    • verify-ca: Weryfikuje certyfikat CA.
    • verify-full: Weryfikuje certyfikat CA i nazwę hosta.
  • debezium.database.sslrootcert - Ścieżka do certyfikatu CA, używanego do weryfikacji serwera PostgreSQL.
  • debezium.database.sslcert - Ścieżka do certyfikatu klienta (PEM), używanego do uwierzytelnienia klienta.
  • debezium.database.sslkey - Ścieżka do klucza prywatnego klienta (PEM).
chmod 600 client-key.pem
chmod 644 client-cert.pem server-ca.pem
  1. Załozenie topikow
  2. Run
nohup ./run.sh > debezium.log 2>&1 &

PUBSUB sik Configuration

Snapshot

snapshot.mode:

  • initial (default) - never performs snapshots

  • always,

  • no_data,

  • when_needed - After the connector starts, it performs a snapshot only if it detects one of the following circumstances:

    • It cannot detect any topic offsets.
    • A previously recorded offset specifies a log position that is not available on the server.

  • configuration_based,

  • custom

Configuration

# PUB/SUB
debezium.sink.type=pubsub
debezium.sink.pubsub.project.id=lab-biz-app-gcp-pr
debezium.sink.pubsub.ordering.enabled=true
debezium.sink.pubsub.null.key=default
#topic.creation.enable=true
# PostgreSQL
debezium.postgres:type=connector-metrics,context=streaming,server=cdc-debezium,database=mydb,table=app.rachunki
debezium.deployment.mode=default
debezium.source.kafka.connect.uris=http://localhost:8083
# PostgreSQL
debezium.source.connector.class=io.debezium.connector.postgresql.PostgresConnector
# trzeba utworzyć plik /data/offsets.dat
debezium.source.offset.storage.file.filename=/data/offsets.dat
debezium.source.offset.flush.interval.ms=0
debezium.source.database.hostname=<postgres instance ip>
debezium.source.database.port=5432
#debezium.source.database.sslmode=verify-full
debezium.source.database.sslmode=verify-ca
debezium.source.database.sslrootcert=/debezium-server/server-ca.pem
debezium.source.database.sslcert=/debezium-server/client-cert.pem
debezium.source.database.sslkey=/debezium-server/client-key.pk8
debezium.source.database.user=replication_user
debezium.source.database.password=*****
debezium.source.database.dbname=mydb
#debezium.source.database.server.name=db-lab-app-gcp-pr
debezium.source.topic.prefix=cdc-debezium
debezium.source.table.include.list=app.karty,app.osoby,app.rachunki
debezium.source.plugin.name=pgoutput
debezium.source.snapshot.mode=never
# quarkus
quarkus.log.console.json=false
#quarkus.log.level=DEBUG
debezium.source.key.converter.schemas.enable=false
debezium.source.value.converter.schemas.enable=false
debezium.source.key.enforce.uniqueness=false
debezium.source.transforms=unwrap,TimestampConverter
debezium.source.time.precision.mode=connect
debezium.source.decimal.handling.mode=string
debezium.source.transforms.TimestampConverter.field=rc_czas
debezium.source.transforms.TimestampConverter.type=org.apache.kafka.connect.transforms.TimestampConverter$Value
debezium.source.transforms.TimestampConverter.format=yyyy-MM-dd HH:mm:ss.SSSSSS
debezium.source.transforms.TimestampConverter.target.type=string
debezium.source.transforms.unwrap.type=io.debezium.transforms.ExtractNewRecordState
debezium.source.transforms.unwrap.drop.tombstones=false
debezium.source.transforms.unwrap.delete.handling.mode=rewrite
debezium.source.transforms.unwrap.add.fields=op:action,schema,table,source.ts_us:timestamp
debezium.source.transforms.unwrap.add.fields.prefix=

Service configuration

/etc/systemd/system/debezium.service

[Unit]
Description=Debezium Server
After=network.target

[Service]
#Type=forking
Type=simple
User=root
Group=root
WorkingDirectory=/debezium-server
ExecStart=/debezium-server/run.sh
#Restart=on-failure
Restart=on-abnormal
[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl start debezium.service
sudo systemctl status debezium.service
sudo journalctl -u debezium.service