Home - niravthakkar/kafka GitHub Wiki

Apache Kafka can be built for Linux on z Systems running RHEL 7.1 or 6.6 or SLES 12 or 11 by following these instructions. Version 0.9.0 has been successfully built & tested this way.

General Notes: i) When following the steps below please use a standard permission user unless otherwise specified.
ii) A directory /<source_root>/ will be referred to in these instructions, this is a temporary writeable directory anywhere you'd like to place it

Building Apache Kafka

Install the build time dependencies

On RHEL 6.6 and RHEL 7.1 systems
```
sudo yum install git wget unzip java-1.7.1-ibm-devel.s390x
```
On SLES 12 systems
```
sudo zypper install git wget unzip java-1_7_1-ibm-devel
```
On SLES 11 systems
```
sudo zypper install git wget unzip java-1_7_0-ibm-devel java-1_7_0-ibm
```
You may already have some of these packages installed - just install any that are missing.
Install gradle (unless you have it already)
```
cd /<source_root>/
wget https://services.gradle.org/distributions/gradle-2.5-bin.zip
unzip gradle-2.5-bin.zip
export PATH=$PATH:/<source_root>/gradle-2.5/bin
```
Other versions of gradle may work, but this has been tested with 2.5
Note: Where <source_root> is the directory defined at the top of this document - if you wish to clean up the <source_root> at the end of the install you may want to place gradle in a different location, if so just update the paths
Download the required version of the Apache Kafka source
```
git clone https://github.com/apache/kafka
cd kafka
git checkout 0.9.0
```
Note: The github location is a mirror of the Apache location https://git-wip-us.apache.org/repos/asf/kafka.git
Modify the build process to use a more recent build of Snappy Java

Snappy Java prior to version 1.1.2 does not support Linux on z Systems, however fixes for the issue have been provided to the community and version 1.1.2 now incorporates an s390 build. However the current release (at the time of writing) of Apache Kafka does not use this later version of Snappy Java, so we update the build process to use it, as below:

Modify the build.gradle file to use a later version
```
vi build.gradle
```
Update the snappy-java version to be 1.1.2 rather than 1.1.1.7 it is as default
```
project(':clients') {
    apply plugin: 'checkstyle'
    archivesBaseName = "kafka-clients"

dependencies {
    compile "$slf4japi"
    compile 'org.xerial.snappy:snappy-java:1.1.2'
    compile 'net.jpountz.lz4:lz4:1.2.0'
```
Configure Java and run the gradle build process

Set the JAVA_HOME environment variable to:
```
export JAVA_HOME=/etc/alternatives/java_sdk_ibm
```
Now setup gradle and build the jar files
```
gradle
gradle jar
```
Optionally use the built in tests to verify Apache Kafka
```
gradle test
```
Note: Occasionally tests will fail,but re-running will normally resolve them - These issues are not Linux on z Systems specific, they are also present on x86 platforms. Test stability issues have been reported here
Optionally use the example scripts to run a single server

Apache Kafka provides a quickstart guide, this has a simple example that is replicated below (with a small patch), head to kafka.apache.org for the guide.
1. Update the -X options
```
cd /<source_root>/kafka/
vi bin/kafka-run-class.sh
```
  The script that starts the zookeeper creates a gc log, however it uses an option that isn't supported on IBM Java (the -X options are VM specific extensions to the Java spec), so we need to update -Xloggc to -Xverbosegclog as below:
```
# GC options
GC_FILE_SUFFIX='-gc.log'
GC_LOG_FILE_NAME=''
if [ "x$GC_LOG_ENABLED" = "xtrue" ]; then
  GC_LOG_FILE_NAME=$DAEMON_NAME$GC_FILE_SUFFIX
  KAFKA_GC_LOG_OPTS="-Xverbosegclog:$LOG_DIR/$GC_LOG_FILE_NAME -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps "
fi
```
2. Now launch the two single node servers
```
bin/zookeeper-server-start.sh -daemon config/zookeeper.properties
bin/kafka-server-start.sh -daemon config/server.properties
```
  Both servers should start in the background, check the logs in /<source_root>/kafka/logs/ for more information.
3. Now create and list a simple test topic and messages file
  
  Please note the quickstart guide refer to bin/kafka-create-topic.sh however this has been replaced with bin/kafka-topics.sh and flags such as --create or --delete and some modifications to the parameters:
```
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partition 1 --topic test
bin/kafka-topics.sh --list --zookeeper localhost:2181
```
  Create a quick messages file (you can also enter it on the console) then produce those messages onto the test topic created earlier
```
echo -e "Congratulations\nThe build is working\n\nWelcome to Apache Kafka with Linux on z Systems" > /tmp/msg.log
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test < /tmp/msg.log
```
4. Run the consumer and check the results
  
  And finally run a consumer to pull these messages back off the node
```
bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning --max-messages 4
```
  You should see the following:
```
Congratulations
The build is working

Welcome to Apache Kafka with Linux on z Systems
Consumed 4 messages
```
  Note: If the producer is run to accept input from the console (without the < /tmp/msg.log part) and the consumer is run without --max-messages 4 from two different terminals you can see response as you enter each line

References

http://kafka.apache.org/
https://github.com/apache/kafka
https://kafka.apache.org/08/quickstart.html