Configure Scala Kinesis Enrich - artsy/snowplow GitHub Wiki

HOME > SNOWPLOW SETUP GUIDE > Step 3: Setting up Enrich > Step 3.2: setting up Scala Kinesis Enrich > 2: Configuring

This documentation is for version 0.2.0 of Scala Kinesis Enrich, which has not yet been released. Documentation for other versions is available:

Version 0.1.0

The Scala Stream Collector has a number of configuration options available.

Download a template configuration file from GitHub: [default.conf] app-conf.

Now open the default.conf file in your editor of choice.

Values that must be configured are:

• enrich.aws.access-key
• enrich.aws.secret-key

You can insert your actual credentials in these fields. Alternatively, if you set both fields to "env", your credentials will be taken from the environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

The enrich.source setting determines which of the supported sources to read raw Snowplow events from:

• "kinesis" for reading Thrift-serialized records from a named Amazon Kinesis stream
• "stdin" for reading Base64-encoded Thrift-serialized records from the app's own stdin I/O stream

If you select "kinesis", you need to set enrich.streams.in to the name of your raw Snowplow event stream configured in your Scala Stream Collector.

The enrich.sink setting determines which of the supported sinks to write enriched Snowplow events to:

• "kinesis" for writing enriched Snowplow events to a named Amazon Kinesis stream
• "stdouterr" for writing enriched Snowplow events records to the app's own stdout I/O stream

If you select "kinesis", you need to also update the enrich.streams.out section:

out: {
  enriched: "SnowplowEnriched"
  enriched_shards: 1 # Number of shards to use if created.
  bad: "SnowplowBad" # Not used until #463
  bad_shards: 1 # Number of shards to use if created.
}

Note that the Scala Kinesis Enrich does not yet support writing out bad rows to a dedicated Kinesis stream - so for now you can ignore those settings and simply configure the enriched and enriched_shards fields.

You may wish to use Snowplow's configurable enrichments. To do this, create a directory of enrichment JSONs. For each configurable enrichment you wish to use, the enrichments directory should contain a .json file with a configuration JSON for that enrichment. When you come to run Scala Kinesis Enrich you can then pass in the filepath to this directory using the --enrichments option.

Sensible default configuration enrichments are available on GitHub: 3-enrich/emr-etl-runner/config/enrichments.

See the documentation on configuring enrichments for details on the available enrichments.

Next: Run Scala Kinesis Enrich