Service: Encore Transfer - EyevinnOSC/community GitHub Wiki

Encore Transfer

Encore Transfer (serviceId eyevinn-encore-transfer, catalog page) is an open source worker service from Eyevinn that listens on a Redis queue for completed SVT Encore transcoding jobs and transfers the output files to a destination, typically an S3 bucket. It is a long-running service instance. License: MIT.

What it can do

  • Pick up completed Encore jobs from a Redis queue and transfer their output to an S3 destination.
  • Act as the transfer stage in an Encore pipeline: Encore (transcode), Valkey (queue), Encore Transfer, and optionally Encore Packager for HLS/DASH packaging.
  • Write each job's output to a subfolder named from the job id under the configured output location.

Creating an instance

Create first and confirm running:

  • An SVT Encore instance (serviceId encore) for transcoding.
  • A Valkey instance (serviceId valkey-io-valkey, see Valkey) used as the job queue, shared with Encore. The catalog links the Valkey TCP address to RedisUrl.

You also need an OSC personal access token and, for AWS S3 output, the AWS credentials. Store sensitive values as secrets with create-service-secret (serviceId: "eyevinn-encore-transfer"), for example oscat, awsaccesskeyid, awssecretaccesskey. Then call create-service-instance:

{
  "name": "mytransfer",
  "RedisUrl": "redis://<valkey-host>:<valkey-port>",
  "RedisQueue": "transfer-queue",
  "Output": "s3://my-bucket/output/",
  "OscAccessToken": "{{secrets.oscat}}",
  "AwsAccessKeyIdSecret": "awsaccesskeyid",
  "AwsSecretAccessKeySecret": "awssecretaccesskey"
}

Get the Valkey address and port from describe-service-instance. Verify with wait-for-service-ready or get-instance-status, and check progress with get-logs-for-instance.

Options

Option Required Type Meaning
name Yes string Name of the instance. Alphanumeric, at most 20 characters.
RedisUrl Yes string, sensitive URL of the Redis server holding the queue, format redis://<host>:<port> (REDIS_URL; upstream default redis://localhost:6379).
RedisQueue No string Name of the Redis queue to listen to (REDIS_QUEUE; upstream default transfer-queue).
Output Yes string S3 bucket and base folder for output, e.g. s3://my-bucket/output/. The actual output goes in a subfolder named from the job id (S3_OUTPUT).
OscAccessToken Yes string, sensitive OSC access token used to reach the Encore instance in OSC (OSC_ACCESS_TOKEN).
AwsAccessKeyIdSecret No string Name of the OSC service secret that holds the AWS access key id for Output (AWS_ACCESS_KEY_ID_SECRET). Per upstream this is a secret name, not the key itself.
AwsSecretAccessKeySecret No string Name of the OSC service secret that holds the AWS secret access key for Output (AWS_SECRET_ACCESS_KEY_SECRET). Per upstream this is a secret name, not the key itself.

{{secrets.NAME}} must be the entire field value. Overwriting a secret does not restart a running instance. The two ...Secret options take the secret's name according to the upstream README; the previous version of this page used the {{secrets.NAME}} form for them, so check with get-service-schema or the logs if credentials are not picked up.

Using the instance

Submit transcoding jobs to your Encore instance. Completed jobs announced on the Redis queue are picked up by Encore Transfer, which moves the output to Output. Use get-logs-for-instance to follow transfers.

Example with the OSC CLI:

npx -y @osaas/cli create eyevinn-encore-transfer mytransfer \
  -o RedisUrl="redis://<valkey-host>:<valkey-port>" \
  -o Output="s3://my-bucket/output/" \
  -o OscAccessToken="{{secrets.oscat}}" \
  -o AwsAccessKeyIdSecret="awsaccesskeyid" \
  -o AwsSecretAccessKeySecret="awssecretaccesskey"

Resources

⚠️ **GitHub.com Fallback** ⚠️