Service: S3 Sync - EyevinnOSC/community GitHub Wiki

Getting Started

With the open web service S3 Sync you can sync files on one S3 bucket to another S3 bucket without having to download all files from one bucket and then upload them to the other bucket. This tutorial help you get started how you can use this service to synchronize two buckets.

Prerequisites

  • If you have not already done so, sign up for an OSC account.
  • A source S3 bucket
  • A destination S3 bucket

Step 1: Setup source credentials

Navigate to the S3 Sync service in the Eyevinn Open Source Cloud web console and select the tab titled "Service Secrets". Click on button "New Secret".

Enter the name of the secret and call it sourceaccesskey and enter the key id for the source S3 bucket. Now create a secret called sourcesecretkeythat will contain the secret credential for the source S3 bucket. If the source S3 bucket is on AWS the sourceaccesskey is the AWS_ACCESS_KEY_ID and the sourcesecretkey the AWS_SECRET_ACCESS_KEY.

Step 2: Setup destination credentials

Repeat the procedures in step 1 but for the destination S3 bucket. You should now have 4 secrets.

Step 3: Create a S3 sync job

To create and start the job to execute the transfer of the files from the source S3 bucket to the destination S3 bucket choose the tab "My S3 Sync jobs" and click on the button "Create Job".

  • Give the job a unique name, for example guide1.
  • Enter the source and destination S3 bucket URLs, e.g. s3://lab-testcontent-output/osc/VINN-11/ s3://test/
  • Enter the source credentials by referring to the secrets we created in step 1.
  • Enter the region of the source S3 bucket if required. This field is optional.
  • Enter the endpoint of the source S3 bucket if required. This field is optional.
  • Enter the destination credentials by referring to the secrets we created in step 2.
  • Enter the region of the destination S3 bucket if required. This field is optional.
  • Enter the endpoint of the destination S3 bucket if required. This field is optional.

Now press the button "Create" to create and start the job.

Step 4: Verify job completed

You can verify that the files have been successfully completed by clicking on the blue document icon. That will open up the logs for the job. It should look something like this.

Additional Documentation