Debezium peerdb fivetan airbyte - ghdrako/doc_snipets GitHub Wiki

Modern cloud data warehouses, such as BigQuery, can handle and transform vast amounts of data using just SQL. This capability has led many to switch from the traditional ETL to extract, load, transform (ELT), where data is first extracted, then loaded, and finally transformed within the warehouse itself. This change means the data warehouse, which is optimized for such tasks, executes the transformation workload instead of specialized ETL tools.

This tools focus on connecting to various data sources to extract data without meddling with its structure. The transformation responsibility is handed over to the data warehouse. Further assisting in this process, tools such as dbt, Dataform, and SQLMesh offer frameworks to help organize and execute data transformations. However, the heavy lifting – the actual data processing – is done by the data warehouse itself.

PeerDB

• https://blog.peerdb.io/benchmarking-postgres-replication-peerdb-vs-airbyte

Airbyte

• https://airbyte.com/
• https://github.com/PacktPublishing/Fundamentals-of-Analytics-Engineering/blob/main/chapter_8/guides/setting_up_airbyte_cloud.md

Fivetan

https://www.fivetran.com/

Debezium

• https://www.baeldung.com/debezium-intro

• https://debezium.io/documentation/reference/2.2/operations/debezium-server.html

• https://debezium.io/documentation/reference/stable/operations/debezium-server.html

• https://medium.com/google-cloud/change-data-capture-with-debezium-server-on-gke-from-cloudsql-for-postgresql-to-pub-sub-d1c0b92baa98

• https://debezium.io/documentation/reference/stable/integrations/serdes.html

• https://blog.devgenius.io/realtime-change-data-capture-to-bigquery-no-kafka-no-dataflow-fb4a6994441b

• https://cloud.google.com/dataflow/docs/guides/templates/provided/mysql-change-data-capture-to-bigquery Full refresh:

• https://stackoverflow.com/questions/57753480/use-debezium-to-capture-the-existing-data-full-refresh?rq=3 Postgres:

• https://debezium.io/documentation/reference/1.4/connectors/postgresql.html

Debezium Server

• https://debezium.io/documentation/reference/operations/debezium-server.html The Debezium server is a configurable, ready-to-use application that streams change events from a source database to a variety of messaging infrastructures.

Postgres

• https://debezium.io/documentation/reference/stable/postgres-plugins.html
• https://debezium.io/documentation/reference/stable/connectors/postgresql.html Debezium’s supported logical decoding plug-ins,
• decoderbufs or
• pgoutput

Debezium sink options:

• https://infinitelambda.com/postgres-cdc-debezium-google-pubsub/ ​​Debezium provides a ready-to-use application that streams change events from a source database to messaging infrastructure like
• Amazon Kinesis,
• Google Cloud Pub/Sub,
• Apache Pulsar or
• Redis (Stream)