Service: Encore Packager - EyevinnOSC/community GitHub Wiki

Encore Packager

Encore Packager (serviceId eyevinn-encore-packager) listens for completed SVT Encore transcoding jobs on a Redis queue and packages their output into adaptive streaming formats (DASH and HLS) using Shaka Packager. It is a long-running service instance under the MIT license, with up to 50 GiB of storage per instance. It is a VOD pipeline and has no live input support. Source: Eyevinn/encore-packager.

What it can do

  • Package the output of Encore transcoding jobs as DASH (manifest.mpd) and HLS (master.m3u8).
  • Write the result to a local path or to an S3-compatible bucket (AWS S3, or for example MinIO).
  • Skip packaging and instead copy MP4 files and generate SMIL playlists (SkipPackaging).
  • Process several jobs in parallel (Concurrency) and send success or failure callbacks (CallbackUrl).

Creating an instance

Create the dependencies first and confirm they are running:

  1. A running SVT Encore instance (serviceId encore) that produces the jobs to package.
  2. Valkey (serviceId valkey-io-valkey) for the job queue. Use describe-service-instance to read its address and form RedisUrl.
  3. Output storage: an S3-compatible bucket, for example a MinIO instance (serviceId minio-minio; note its endpoint, access key and secret key), or an existing AWS S3 bucket with an access key ID and secret.

Store credentials as service secrets with create-service-secret on serviceId eyevinn-encore-packager, for example oscpat (your OSC personal access token, used to access Encore instances in OSC), awskey (AWS or MinIO access key ID) and awssecret (secret access key). Then call create-service-instance with serviceId: "eyevinn-encore-packager":

{
  "name": "mypackager",
  "RedisUrl": "redis://<valkey-host>:6379",
  "OutputFolder": "s3://my-output-bucket",
  "PersonalAccessToken": "{{secrets.oscpat}}",
  "AwsAccessKeyId": "{{secrets.awskey}}",
  "AwsSecretAccessKey": "{{secrets.awssecret}}",
  "AwsRegion": "eu-west-1"
}

For MinIO or another non-AWS store also set S3EndpointUrl. Wait for the instance to be running with wait-for-service-ready and check get-logs-for-instance to see it connect to the queue.

CLI equivalent:

osc create eyevinn-encore-packager mypackager \
  -o RedisUrl="redis://<valkey-host>:6379" \
  -o OutputFolder="s3://my-output-bucket" \
  -o PersonalAccessToken="{{secrets.oscpat}}" \
  -o AwsAccessKeyId="{{secrets.awskey}}" \
  -o AwsSecretAccessKey="{{secrets.awssecret}}" \
  -o AwsRegion="eu-west-1"

Options

Option Required Type Meaning
name Yes string Name of the instance. Alphanumeric, at most 20 characters.
RedisUrl Yes string, sensitive URL of the Redis server used for message queuing when running as a service. Default redis://localhost:6379. The packager listens for packaging job messages on it.
RedisQueue No string Name of the Redis queue to listen to for packaging job messages. Default packaging-queue.
OutputFolder Yes string Base folder for packaging output; the actual output goes in subfolders according to OutputSubfolderTemplate. Default packaged. A local path or an S3 bucket such as s3://my-output-bucket.
Concurrency No string Number of packaging jobs processed in parallel. Default 1.
PersonalAccessToken Yes string, sensitive OSC access token used to access Encore instances hosted in OSC.
AwsAccessKeyId Yes string, sensitive Access key ID for authentication when OutputFolder is an S3 bucket (AWS or MinIO). Used together with AwsSecretAccessKey.
AwsSecretAccessKey Yes string, sensitive Secret access key for authentication when OutputFolder is an S3 bucket.
AwsRegion No string AWS region for S3 operations, for example eu-west-1, when OutputFolder is an S3 bucket.
AwsSessionToken No string, sensitive Session token for temporary AWS credentials, typically with IAM roles or STS.
S3EndpointUrl No string Custom S3 endpoint URL when the bucket is not hosted on AWS, for example MinIO, DigitalOcean Spaces or another S3-compatible provider.
OutputSubfolderTemplate No string Subfolder structure relative to OutputFolder. Default $INPUTNAME$/$JOBID$. The keywords $INPUTNAME$, $JOBID$ and $EXTERNALID$ are replaced with the basename of the input, the Encore job ID and the external ID.
SkipPackaging No boolean When true, copies the MP4 files and creates a SMIL file instead of producing DASH/HLS with Shaka Packager. Default false.
CallbackUrl No string URL of a callback service that receives HTTP notifications when packaging succeeds or fails. Basic authentication is supported as https://user:password@hostname/path.

Sensitive options (RedisUrl, PersonalAccessToken, AwsAccessKeyId, AwsSecretAccessKey, AwsSessionToken) should be service secrets passed as {{secrets.NAME}} as the whole field value. Changing a secret later does not restart a running instance. The AWS key options are described upstream as needed when writing to an S3 bucket.

Using the instance

Encore Packager subscribes to the Valkey queue and waits for job completion messages from SVT Encore. For each job it:

  1. Fetches the transcoded files from the Encore output location.
  2. Runs Shaka Packager to produce the DASH manifest manifest.mpd and HLS playlist master.m3u8.
  3. Writes the packaged output to OutputFolder/<InputName>/<JobId>/ (with the default subfolder template).
  4. Sends the optional callback on success or failure.

Resources

⚠️ **GitHub.com Fallback** ⚠️