GCP Connect other providers - ghdrako/doc_snipets GitHub Wiki

Resource:

Data exchange between Google Cloud and other CSPs

For transferring data to Cloud Storage or BigQuery, you can also use Storage transfer service or BigQuery transfer service.

Storage Transfer Service

Cloud Storage Transfer Service provides us with the option to transfer or back up our data from online sources to a data sink. For example, that we want to move data from our AWS S3 bucket into our Cloud Storage bucket. S3 would be our source and the destination Cloud Storage bucket would be our data sink. We can also use Cloud Storage Transfer Service to move data from an HTTP/HTTP(s) source or even another Cloud Storage bucket. To make these data migrations or synchronizations easier, Cloud Storage Transfer Service offers several options:

  • We can schedule a one-time transfer operation or a recurring operation.
  • We can delete existing objects in the destination data sink if they don't have any corresponding objects in the source.
  • We can delete source objects once we have transferred them.
  • We can schedule periodic synchronizations between the source and destination using folders based on file creation dates, filenames, or even the time of day you wish to import data.

You can also use gsutil to transfer data between Cloud Storage and other locations. You can speed up the process of transferring files on-premises by using the -m option to enable a multi-threaded copy. If you plan to upload larger files, gsutil will split these files into several smaller chunks and upload these in parallel. It is recommended that you use Cloud Storage Transfer Service when transferring data from Amazon S3 to Cloud Storage.

BigQuery Data Transfer Service

The BigQuery Data Transfer Service automates data movement into BigQuery on a scheduled, managed basis. Your analytics team can lay the foundation for a BigQuery data warehouse without writing a single line of code.

it cannot be used for transferring data out of BigQuery

You can access the BigQuery Data Transfer Service using the:

Supported data sources: