rclone - yar145/mytestrepo1 GitHub Wiki
You can set up rclone with Google Drive in an unattended mode, i.e. not tied to a specific end-user Google account. This is useful when you want to synchronise files onto machines that don't have actively logged-in users, for example build machines.
To use a Service Account instead of OAuth2 token flow, enter the path to your Service Account credentials at the service_account_file prompt during rclone config and rclone won't use the browser based authentication flow. If you'd rather stuff the contents of the credentials file into the rclone config file, you can set service_account_credentials with the actual contents of the file instead, or set the equivalent environment variable.
Use case - Google Apps/G-suite account and individual Drive Let's say that you are the administrator of a Google Apps (old) or G-suite account. The goal is to store data on an individual's Drive account, who IS a member of the domain. We'll call the domain example.com, and the user [email protected].
There's a few steps we need to go through to accomplish this:
- Create a service account for example.com To create a service account and obtain its credentials, go to the Google Developer Console. You must have a project - create one if you don't. Then go to "IAM & admin" -> "Service Accounts". Use the "Create Service Account" button. Fill in "Service account name" and "Service account ID" with something that identifies your client. Select "Create And Continue". Step 2 and 3 are optional. These credentials are what rclone will use for authentication. If you ever need to remove access, press the "Delete service account key" button.
- Allowing API access to example.com Google Drive Go to example.com's admin console Go into "Security" (or use the search bar) Select "Show more" and then "Advanced settings" Select "Manage API client access" in the "Authentication" section In the "Client Name" field enter the service account's "Client ID" - this can be found in the Developer Console under "IAM & Admin" -> "Service Accounts", then "View Client ID" for the newly created service account. It is a ~21 character numerical string. In the next field, "One or More API Scopes", enter https://www.googleapis.com/auth/drive to grant access to Google Drive specifically.
For google to google you can make use of server-side copying and you won't even need to touch the data locally.
There is two ways to enable server-side copying. Either: Set this in your config file under your drive remote: server_side_across_configs = true or... add this flag to whatever rclone command you run --drive-server-side-across-configs true Either is fine - no need to do both.
Then you can simply use rclone copy (or rclone move if you prefer) and rclone will server-side the operation if possible. rclone copy OldGdrive: NewGdrive:\FromOldDrive -P (-P here will enable some statistics to let you see the progress)
Server-side copies have nearly unlimited transfer speed, so they are very fast - but you will have to tell the server each file to copy, so the copy-time will mostly be determined by the sheer amount of files you have. The size of the files won't matter so much when you can copy them at gigabits pr second.
Using a VM in CGP like you mentioned is something that is often done, but usually only if you can't server-side it normally for whatever reason. For example if you need to import data between two completely different service-providers - or you not only need to transfer but also encrypt the data ect. In that case you don't need some huge storage in the VM no - a GCP microinstance will do, which would in practical terms be free for at least your first year (since you get free credits to use), but with careful use it would it would be free even past then - since there are certain things you can do within certain limits that remain free.
Ok, so here is how I just did a sucessful test between two of my regular free private Gdrvies:
On Drive1, using the googleUI, right-click on a folder (let's call this "testshare" for this example) and select share. Share it to the email address connected to your Drive2.
On Drive2, we now want to make a local copy of the stuff that got shared, and we can access this with by using --drive-shared-with-me. Let's use the following command:
rclone copy Drive2: --drive-shared-with-me Drive2:\localcopy -v -P (I don't think you need to explicitly enable server-side copy in this sort of setup)
Business drive to another business drive in different domain
Impersonate for Shared Drives You can set it using the RCLONE_DRIVE_TEAM_DRIVE 7 variable.
However you may prefer to set it just on the remote. Let's say your remote is called tsp_shared_dest:.
You can use
tsp_shared_dest,team_drive=XXXXX: to access the team drive and this has the advantage it doesn't affect any other google drive remotes on the command line.
You can also use RCLONE_CONFIG_tsp_shared_dest_TEAM_DRIVE=XXX if you prefer.
export RCLONE_CONFIG_tsp_shared_dest_IMPERSONATE=[email protected]
Preparation for Google Workspace by enabling domain-wide delegation using OAuth servic