Kafka Connect Elasticsearch setup - ScreamingUdder/wiki GitHub Wiki

For setting up Kafka Connect for the Elastic Stack (ELK)

Kafka connect documentation

Confluent blog

N.B. Confluent 3.0.1 claims to include the Elasticsearch connector, but I could not get it to work or find any evidence that it is actually there.

Steps

Installed ELK:

# install elasticsearch + kibana
wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
echo "deb https://packages.elastic.co/elasticsearch/5.x/debian stable main" | sudo tee -a /etc/apt/sources.list
echo "deb https://packages.elastic.co/kibana/5.0.0-alpha/debian stable main" | sudo tee -a /etc/apt/sources.list
sudo apt-get update
sudo apt-get -y install elasticsearch kibana

Started services:

sudo /bin/systemctl daemon-reload
sudo systemctl start elasticsearch.service
sudo systemctl start kibana.service

Downloaded Kafka (which contains Kafka Connect):

wget http://apache.mirror.anlx.net/kafka/0.9.0.1/kafka_2.11-0.9.0.1.tgz
tar -zxf kafka-0.9.0.1-src.tgz

Downloaded kafka-connect-elasticsearch:

wget https://github.com/confluentinc/kafka-connect-elasticsearch/archive/v3.1.0-rc1.tar.gz
tar -xzf v3.1.0-rc1.tar.gz

Edit connect-standalone.properties and quickstart-elasticsearch.properties to correct host and topic names. Note, if the topic name has caps then you'll need to map it to a lowercase name in quickstart-elasticsearch.properties or elasticsearch will be upset, for example:

topic.index.map=TOPIC_NAME_IN_CAPS:topic_name_in_lowercase
topic.key.ignore = TOPIC_NAME_IN_CAPS

Start elasticsearch and kibana:

sudo systemctl start elasticsearch
sudo systemctl start kibana

There should be pages available on localhost:9200 (elasticsearch) and localhost:5601 (kibana). Start the connector with:

export CLASSPATH=/path/to/elasticsearch/connector/*
bin/connect-standalone.sh config/connect-standalone.properties connector1.properties