Service: Player Analytics Worker - EyevinnOSC/community GitHub Wiki

Player Analytics Worker

Player Analytics Worker (serviceId eyevinn-player-analytics-worker) is part of the Eyevinn Player Analytics Specification (EPAS), an open source and open standard for video player analytics. The worker picks up events that the Player Analytics Eventsink wrote to an SQS queue and writes them to a ClickHouse server for further processing according to EPAS. In Eyevinn Open Source Cloud (OSC) it is a long-running service instance. Source: player-analytics-worker.

What it can do

  • Poll an SQS queue for video player analytics events.
  • Write the events to ClickHouse; it creates the database if it is missing.
  • Scale processing with several worker processes (NumWorkers) and batched queue reads (BatchSize).
  • Work with SQS-compatible queues outside AWS, such as SmoothMQ, through SqsEndpoint.

Creating an instance

The worker needs a queue (filled by the Eventsink) and a ClickHouse server. Create these first and confirm they are running:

  1. Player Analytics Eventsink (eyevinn-analytics-eventsink) with its SQS queue, see Player Analytics Eventsink and SmoothMQ (poundifdef-smoothmq). The eventsink pushes the events it receives to the queue.
  2. ClickHouse Server (clickhouse-clickhouse), see ClickHouse. Create it with a name, a database (for example event_db), a user and a password. Prefer strong credentials stored as service secrets.

Then collect the values for the worker:

  • ClickHouseUrl: the ClickHouse instance URL with credentials added, in the form https://<user>:<password>@<instance-host>. Read the host with describe-service-instance.
  • SqsQueueUrl: the URL of the queue. With an SQS-compatible endpoint it can be obtained by creating the queue (idempotent) with the AWS CLI: AWS_ACCESS_KEY_ID=<key> AWS_SECRET_ACCESS_KEY=<secret> AWS_REGION=osc aws sqs create-queue --queue-name=events --endpoint-url=<SqsEndpoint>, which returns for example https://sqs.us-east-1.amazonaws.com/1/events.
  • AwsAccessKeyId and AwsSecretAccessKey: the access key and secret key you configured when setting up the SmoothMQ queue.
  • SqsEndpoint: the URL of the SmoothMQ instance.

Because a {{secrets.NAME}} reference must be the whole field value, store ClickHouseUrl (it embeds the password), AwsAccessKeyId and AwsSecretAccessKey as service secrets with create-service-secret (serviceId: "eyevinn-player-analytics-worker"). Then create the instance with create-service-instance. The instance name must be alphanumeric and at most 20 characters.

{
  "name": "playeranalyticsworker",
  "ClickHouseUrl": "{{secrets.clickhouseurl}}",
  "SqsQueueUrl": "https://sqs.us-east-1.amazonaws.com/1/events",
  "AwsAccessKeyId": "{{secrets.awskey}}",
  "AwsSecretAccessKey": "{{secrets.awssecret}}",
  "SqsEndpoint": "https://<smoothmq-instance-host>"
}

Verify with wait-for-service-ready or get-instance-status, and check get-logs-for-instance if events do not arrive.

Options

Option Required Type Meaning
name Yes string Name of the instance. Letters, digits and underscore only; on OSC at most 20 characters.
ClickHouseUrl Yes string Connection URL of the ClickHouse database where events are stored, including credentials: https://<username>:<password>@<instance-host>. Contains a password, so store it as a service secret.
SqsQueueUrl Yes string URL of the SQS queue the worker polls for events, for example https://sqs.us-east-1.amazonaws.com/1/events.
AwsAccessKeyId Yes string, sensitive Access key id used to authenticate with SQS when reading from the queue.
AwsSecretAccessKey Yes string, sensitive Secret access key used to authenticate with SQS.
SqsEndpoint No string Custom SQS endpoint URL for SQS-compatible services outside standard AWS regions, for example the URL of a SmoothMQ instance.
NumWorkers No string Number of worker processes spawned to process events from the queue (concurrency). Value is a number given as a string.
BatchSize No string Maximum number of messages fetched from the queue in one batch. Value is a number given as a string.

Using the instance

Events are stored in ClickHouse, which you can query and create tables in. Open the ClickHouse Server instance application (the ClickHouse query UI) and sign in with the user and password you created for the database.

Resources

⚠️ **GitHub.com Fallback** ⚠️