Service: Player Analytics Worker - EyevinnOSC/community GitHub Wiki
Player Analytics Worker (serviceId eyevinn-player-analytics-worker) is part of the Eyevinn Player Analytics Specification (EPAS), an open source and open standard for video player analytics. The worker picks up events that the Player Analytics Eventsink wrote to an SQS queue and writes them to a ClickHouse server for further processing according to EPAS. In Eyevinn Open Source Cloud (OSC) it is a long-running service instance. Source: player-analytics-worker.
- Poll an SQS queue for video player analytics events.
- Write the events to ClickHouse; it creates the database if it is missing.
- Scale processing with several worker processes (
NumWorkers) and batched queue reads (BatchSize). - Work with SQS-compatible queues outside AWS, such as SmoothMQ, through
SqsEndpoint.
The worker needs a queue (filled by the Eventsink) and a ClickHouse server. Create these first and confirm they are running:
- Player Analytics Eventsink (
eyevinn-analytics-eventsink) with its SQS queue, see Player Analytics Eventsink and SmoothMQ (poundifdef-smoothmq). The eventsink pushes the events it receives to the queue. - ClickHouse Server (
clickhouse-clickhouse), see ClickHouse. Create it with a name, a database (for exampleevent_db), a user and a password. Prefer strong credentials stored as service secrets.
Then collect the values for the worker:
-
ClickHouseUrl: the ClickHouse instance URL with credentials added, in the formhttps://<user>:<password>@<instance-host>. Read the host withdescribe-service-instance. -
SqsQueueUrl: the URL of the queue. With an SQS-compatible endpoint it can be obtained by creating the queue (idempotent) with the AWS CLI:AWS_ACCESS_KEY_ID=<key> AWS_SECRET_ACCESS_KEY=<secret> AWS_REGION=osc aws sqs create-queue --queue-name=events --endpoint-url=<SqsEndpoint>, which returns for examplehttps://sqs.us-east-1.amazonaws.com/1/events. -
AwsAccessKeyIdandAwsSecretAccessKey: the access key and secret key you configured when setting up the SmoothMQ queue. -
SqsEndpoint: the URL of the SmoothMQ instance.
Because a {{secrets.NAME}} reference must be the whole field value, store ClickHouseUrl (it embeds the password), AwsAccessKeyId and AwsSecretAccessKey as service secrets with create-service-secret (serviceId: "eyevinn-player-analytics-worker"). Then create the instance with create-service-instance. The instance name must be alphanumeric and at most 20 characters.
{
"name": "playeranalyticsworker",
"ClickHouseUrl": "{{secrets.clickhouseurl}}",
"SqsQueueUrl": "https://sqs.us-east-1.amazonaws.com/1/events",
"AwsAccessKeyId": "{{secrets.awskey}}",
"AwsSecretAccessKey": "{{secrets.awssecret}}",
"SqsEndpoint": "https://<smoothmq-instance-host>"
}Verify with wait-for-service-ready or get-instance-status, and check get-logs-for-instance if events do not arrive.
| Option | Required | Type | Meaning |
|---|---|---|---|
| name | Yes | string | Name of the instance. Letters, digits and underscore only; on OSC at most 20 characters. |
| ClickHouseUrl | Yes | string | Connection URL of the ClickHouse database where events are stored, including credentials: https://<username>:<password>@<instance-host>. Contains a password, so store it as a service secret. |
| SqsQueueUrl | Yes | string | URL of the SQS queue the worker polls for events, for example https://sqs.us-east-1.amazonaws.com/1/events. |
| AwsAccessKeyId | Yes | string, sensitive | Access key id used to authenticate with SQS when reading from the queue. |
| AwsSecretAccessKey | Yes | string, sensitive | Secret access key used to authenticate with SQS. |
| SqsEndpoint | No | string | Custom SQS endpoint URL for SQS-compatible services outside standard AWS regions, for example the URL of a SmoothMQ instance. |
| NumWorkers | No | string | Number of worker processes spawned to process events from the queue (concurrency). Value is a number given as a string. |
| BatchSize | No | string | Maximum number of messages fetched from the queue in one batch. Value is a number given as a string. |
Events are stored in ClickHouse, which you can query and create tables in. Open the ClickHouse Server instance application (the ClickHouse query UI) and sign in with the user and password you created for the database.