GCP Connect other providers - ghdrako/doc_snipets GitHub Wiki
Resource:
Data exchange between Google Cloud and other CSPs
- Transfer of data through public IP addresses over the internet.
- Using a managed VPN service between Google Cloud and the other CSP.
- Using Partner Interconnect on Google Cloud to connect to a partner that can directly connect with the other CSP.
- Using Dedicated Interconnect on Google Cloud to transfer data to the other CSP through a common point of presence (POP).
For transferring data to Cloud Storage or BigQuery, you can also use Storage transfer service or BigQuery transfer service.
Storage Transfer Service
Cloud Storage Transfer Service provides us with the option to transfer or back up our data from online sources to a data sink. For example, that we want to move data from our AWS S3 bucket into our Cloud Storage bucket. S3 would be our source and the destination Cloud Storage bucket would be our data sink. We can also use Cloud Storage Transfer Service to move data from an HTTP/HTTP(s) source or even another Cloud Storage bucket. To make these data migrations or synchronizations easier, Cloud Storage Transfer Service offers several options:
- We can schedule a one-time transfer operation or a recurring operation.
- We can delete existing objects in the destination data sink if they don't have any corresponding objects in the source.
- We can delete source objects once we have transferred them.
- We can schedule periodic synchronizations between the source and destination using folders based on file creation dates, filenames, or even the time of day you wish to import data.
You can also use gsutil to transfer data between Cloud Storage and other locations. You can speed up the process of transferring files on-premises by
using the -m option to enable a multi-threaded copy. If you plan to upload larger files, gsutil will split these files into several smaller chunks and upload these in parallel. It is recommended that you use Cloud Storage Transfer Service when transferring data from Amazon S3 to Cloud Storage.
BigQuery Data Transfer Service
The BigQuery Data Transfer Service automates data movement into BigQuery on a scheduled, managed basis. Your analytics team can lay the foundation for a BigQuery data warehouse without writing a single line of code.
it cannot be used for transferring data out of BigQuery
You can access the BigQuery Data Transfer Service using the:
Supported data sources:
- Google Software as a Service (SaaS) apps
- External cloud storage providers
- Amazon S3 Data warehouses
- Teradata
- Amazon Redshift