Datastream - bobbae/gcp GitHub Wiki

Datastream is a serverless and easy-to-use change data capture (CDC) and replication service. It allows you to synchronize data across heterogeneous databases and applications reliably, and with minimal latency and downtime.

Datastream supports streaming from Oracle and MySQL databases into Cloud Storage. The service offers streamlined integration with Dataflow templates to power up-to-date materialized views in BigQuery for analytics, replicate your databases into Cloud SQL or Cloud Spanner for database synchronization, or leverage the event stream directly from Cloud Storage to realize event-driven architectures.

Concepts

https://cloud.google.com/datastream/docs/concepts

Use cases

  • For analytics use Datastream with a pre-built Dataflow template to create up-to-date replicated tables in BigQuery in a fully-managed way.
  • For database replication use Datastream with pre-built Dataflow templates to continuously replicate and synchronize database data into Cloud SQL for PostgreSQL or Spanner to power low-downtime database migration or hybrid-cloud configuration.
  • For building event-driven architectures use Datastream to ingest changes from multiple sources into object stores like Google Cloud Storage or, in the future, messaging services such as Pub/Sub or Kafka
  • Streamline real-time data pipeline that continually streams data from legacy relational data stores (like Oracle and MySQL) using Datastream into MongoDB.

Change Data Capture

https://cloud.google.com/blog/products/data-analytics/serverless-change-data-capture-and-replication

Implementing Change Data Capture using GCP Datastream

https://poojakelgaonkar.medium.com/implementing-change-data-capture-using-gcp-datastream-c340238b5d2b

Using Datastream into MongoDB

https://cloud.google.com/blog/products/databases/get-real-time-analytics-data-with-datastream-and-mongodb

CDC with Striim

https://cloud.google.com/blog/products/data-analytics/data-integration-from-oracle-to-google-bigquery-using-striim/

Datastream How-to Guides

https://cloud.google.com/datastream/docs/how-to

Quickstart

https://cloud.google.com/datastream/docs/quickstart

Tutorials

https://cloud.google.com/datastream/docs/tutorials

Data extraction and ingestion

https://medium.com/google-cloud/what-a-beautiful-stream-40319ac3aaff

Datastream APIs

https://cloud.google.com/datastream/docs/using-datastream-apis

Datastream for BigQuery

https://cloud.google.com/blog/topics/developers-practitioners/golangs-databasesql-driver-support-cloud-spanner-now-generally-available