Run Kafka locally on Docker (Mac only)

Step by step manual on how to run Kafka with Docker on your Mac.

Prerequisite

I'll be using only Docker Toolbox. You can run Docker Toolbox and Docker for Mac at the same time, but that probably will cause you some headache. Just remove Docker for Mac if you already have it installed before you start. For more information about this issue check out this.

You will need to install VirtualBox in case you don't have it already installed. Download VirtualBox here

Install Docker Toolbox using Homebrew

Just run this
brew cask install docker-toolbox.

You either can download it as pkg (here)[https://docs.docker.com/docker-for-mac/docker-toolbox/] or just use Homebrew.

brew install caskroom/cask/docker-toolbox

Now when you run docker-machine ls the output should be something like:

NAME      ACTIVE   DRIVER       STATE     URL                         SWARM   DOCKER        ERRORS
default   -        virtualbox   Running   tcp://192.168.99.102:2376           v17.05.0-ce

If that wasn't the case, just remove your default machine and create a new one.

docker-machine rm default 
docker-machine create --driver virtualbox default

Set docker machine environment variables

Make sure the state of your default Docker machine is Running otherwise you can star it like docker-machine start. Use docker-machine ls to check the state.

Now we need to setup the environment variables so docker will not get lost. Run docker-machine env to see the environment variables of the current running machine. The output must look like this

export DOCKER_TLS_VERIFY="1"
export DOCKER_HOST="tcp://192.168.99.102:2376"
export DOCKER_CERT_PATH="/Users/Amer/.docker/machine/machines/default"
export DOCKER_MACHINE_NAME="default"
# Run this command to configure your shell:
# eval $(docker-machine env)

You can add this to your eval $(docker-machine env) to your what every rc file you have. If you are using bash, this will be in ~/.bashrc or ~/.zshrc for Zsh. (Don't forget to reload your shell!)

If everything went right, when you run docker version command you should see something like:

Client:
 Version:      17.05.0-ce
 API version:  1.29
 Go version:   go1.7.5
 Git commit:   89658be
 Built:        Fri May  5 15:36:11 2017
 OS/Arch:      darwin/amd64

Server:
 Version:      17.05.0-ce
 API version:  1.29 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   89658be
 Built:        Thu May  4 21:43:09 2017
 OS/Arch:      linux/amd64
 Experimental: false

Run Kafka

For this, I'll be using Spotify's container for Kafka on Github.

Now, to start Kafka, all that you need is to run this line:

docker run -p 2181:2181 -p 9092:9092 --env ADVERTISED_HOST=`docker-machine ip \`docker-machine active\`` --env ADVERTISED_PORT=9092 spotify/kafka

If you got a similar output that means Kafka and Zookeeper are running happily.

INFO supervisord started with pid 1
2017-06-12 16:27:47,604 INFO spawned: 'zookeeper' with pid 7
2017-06-12 16:27:47,605 INFO spawned: 'kafka' with pid 8
2017-06-12 16:27:48,629 INFO success: zookeeper entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-06-12 16:27:48,629 INFO success: kafka entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

Important notes

It's right that your Kafka is running locally, but you can't connect to it using localhost or 127.0.0.1 IP. What you need is to find out what is the IP your docker-machine is using. To do that run docker-machine ip. That will return the IP address of the default Docker machine.

Kafka consumer/producer code example with Ruby

Just to give a feeling on how things work, check out this code example written in Ruby.