Data Fusion - bobbae/gcp GitHub Wiki

Data Fusion is a fully managed, cloud-native, scalable data integration platform. It is a GUI based data integration service for building and managing data pipelines.

https://medium.com/google-cloud/designing-a-data-lake-on-gcp-with-data-fusion-and-composer-e2ea0a753525

Data Fusion is built using open source project CDAP, and this open core ensures data pipeline portability for users. CDAP’s broad integration with on-premises and public cloud platforms gives Cloud Data Fusion users the ability to break down silos and deliver insights that were previously inaccessible.

https://cloud.google.com/data-fusion

CDAP is an integrated, open source application development platform for the Hadoop ecosystem that provides developers with data and application abstractions to simplify and accelerate application development, address a broader range of real-time and batch use cases, and deploy applications into production while satisfying enterprise requirements.

https://github.com/cdapio/cdap

Under the hood, Data Fusion executes pipelines on a Dataproc cluster. Data Fusion automatically converts GUI based pipelines into Dataproc jobs for execution whenever a pipeline is executed. It supports two execution engine options: MapReduce and Apache Spark.

CDAP Example Apps

https://github.com/cdapio/cdap-apps

Modern, more secure data lakes on Google Cloud

Cloud Data Fusion helps users build scalable, distributed data lakes on Google Cloud by integrating data from siloed on-premises platforms. Customers can leverage the scale of the cloud to centralize data and drive more value out of their data as a result.

https://towardsdatascience.com/building-a-data-lake-on-gcp-with-cdap-6271c264f22e

The self-service capabilities of Cloud Data Fusion increase process visibility and lower the overall cost of operational support.

https://cloud.google.com/data-fusion/plugins

Choosing Dataflow, Dataprep, Dataproc or Data Fusion

https://wisdomplexus.com/blogs/dataproc-vs-dataflow-vs-dataprep/

Wrangler UI

https://datadice.medium.com/the-wrangler-user-interface-of-cloud-data-fusion-21e96cfe1b2d

Agile data warehouses with BigQuery

Cloud Data Fusion can help organizations better understand their customers by breaking down data silos and enabling development of agile, cloud-based data warehouse solutions in BigQuery. A trusted, unified view of customer engagement and behavior unlocks the ability to drive a better customer experience, which leads to higher retention and higher revenue per customer.

https://cloud.google.com/data-fusion/docs/tutorials/replicating-data/mysql-to-bigquery

Unified analytics environment

Many users today want to establish a unified analytics environment across a myriad of expensive, on-premises data marts. Employing a wide range of disconnected tools and stop-gap measures creates data quality and security challenges. Cloud Data Fusion’s vast variety of connectors, visual interfaces, and abstractions centered around business logic helps in lowering TCO, promoting self-service and standardization, and reducing repetitive work.