Kinesis LZO S3 sink - OXYGEN-MARKET/oxygen-market.github.io GitHub Wiki

HOME > SNOWPLOW TECHNICAL DOCUMENTATION > Storage > Kinesis LZO S3 Sink

Overview

The Kinesis LZO S3 Sink reads records from an Amazon Kinesis stream. The records are serialized in Protocol Buffers arrays using Elephant Bird's RawBlockWriter class, then compressed using [splittable LZO]. This generates .lzo files (the compressed data) and small .lzo.index files (containing byte offsets for the LZO blocks in the corresponding .lzo files, allowing the blocks to be processed in parallel).

See also the setup guide.

Home | About | Project | Setup Guide | Technical Docs | Copyright © 2012-2017 Snowplow Analytics Ltd

⚠️ **GitHub.com Fallback** ⚠️