Configure Scala Kinesis Enrich v0.1 - OXYGEN-MARKET/oxygen-market.github.io GitHub Wiki

HOME > SNOWPLOW SETUP GUIDE > Step 3: Setting up Enrich > Step 3.2: setting up Scala Kinesis Enrich > 2: Configuring

This page refers to version 0.1.0 of Scala Kinesis Enrich. Documentation for other versions is available:

Version 0.2(not yet released)

Scala Kinesis Enrich has a number of configuration options available.

Download a template configuration file from GitHub: config.hocon.sample.

Now open the config.hocon.sample file in your editor of choice.

Values that must be configured are:

• enrich.aws.access-key
• enrich.aws.secret-key

The enrich.source setting determines which of the supported sources to read raw Snowplow events from:

• "kinesis" for reading Thrift-serialized records from a named Amazon Kinesis stream
• "stdin" for reading Base64-encoded Thrift-serialized records from the app's own stdin I/O stream

If you select "kinesis", you need to set enrich.streams.in to the name of your raw Snowplow event stream configured in your Scala Stream Collector.

The enrich.sink setting determines which of the supported sinks to write enriched Snowplow events to:

• "kinesis" for writing enriched Snowplow events to a named Amazon Kinesis stream
• "stdouterr" for writing enriched Snowplow events records to the app's own stdout I/O stream

If you select "kinesis", you need to also update the enrich.streams.out section:

out: {
  enriched: "SnowplowEnriched"
  enriched_shards: 1 # Number of shards to use if created.
  bad: "SnowplowBad" # Not used until #463
  bad_shards: 1 # Number of shards to use if created.
}

Note that the Scala Kinesis Enrich does not yet support writing out bad rows to a dedicated Kinesis stream - so for now you can ignore those settings and simply configure the enriched and enriched_shards fields.

Next, make sure that the enrich.enrichments.geo_ip.maxmind_file configuration points to the MaxMind GeoLiteCity file you downloaded during Installation:

  enrichments {
    geo_ip: {
      enabled: true # false not yet suported
      maxmind_file: "/path/to/downloaded/GeoLiteCity.dat"
    }

Next: Run Scala Kinesis Enrich