Build and run Kafka from source code - tenji/ks GitHub Wiki

Kafka 源码编译和运行

当前编译和运行的 Kafka 版本为 0.10.3，其它版本类似。

一、安装必要软件

1.1 Java 8

C:\Users\s00313265>java -version
java version "1.8.0_45"
Java(TM) SE Runtime Environment (build 1.8.0_45-b15)
Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode)

1.2 Gradle

C:\Users\s00313265>gradle -v

------------------------------------------------------------
Gradle 5.0
------------------------------------------------------------

Build time:   2018-11-26 11:48:43 UTC
Revision:     7fc6e5abf2fc5fe0824aec8a0f5462664dbcd987

Kotlin DSL:   1.0.4
Kotlin:       1.3.10
Groovy:       2.5.4
Ant:          Apache Ant(TM) version 1.9.13 compiled on July 10 2018
JVM:          1.8.0_45 (Oracle Corporation 25.45-b02)
OS:           Windows 7 6.1 amd64

1.3 Scala

C:\Users\s00313265>scala -version
Scala code runner version 2.11.12 -- Copyright 2002-2017, LAMP/EPFL

1.4 Git

C:\Users\s00313265>git --version
git version 2.22.0.windows.1

1.5 IntelliJ IDEA

二、下载源码

  1. Create a new github account, if you don’t have one already
  2. Fork Apache Kafka github project
  3. Using git, clone your forked project in your local m/c
> git clone https://github.com/<your github id>/kafka.git

三、编译源码

在控制台输入以下命令：

  1. Windows：
> cd kafka
> gradle
> .\gradlew.bat jar
  1. Linux：
> cd kafka
> gradle
> ./gradlew jar

四、将工程导入 Intellij IDEA

  1. 选择菜单栏的 File -> Open
  2. 在打开的窗口选择对应 Gradle 项目的build.gradle文件，然后点击 OK
  3. 在 Open Project 窗口中，点击 Open as Project
  4. 在 Import from Gradle 窗口，配置你的 Gradle 项目，然后点击 OK

导入之后会自动构建：

如果遇到以下 Gradle 构建报错：

> You can't map a property that does not exist: propertyName=testClassesDir

参考 KAFKA-7706，编辑 build.gradle 文件，修改org.scoverage:gradle-scoverage版本由 2.1.0 为 2.5.0 ：

classpath 'org.scoverage:gradle-scoverage:2.5.0'

五、启动 ZooKeeper

因为 Kafka 需要依赖 ZooKeeper，因此需要先运行 ZooKeeper，在控制台执行以下命令

  1. Windows：
> cd kafka
> bin\windows\zookeeper-server-start.bat config/zookeeper.properties
  1. Linux：
> cd kafka
> bin/zookeeper-server-start.sh config/zookeeper.properties

六、Debug/Run Kafka in IntelliJ IDEA

  1. In Intellij, click on Run > Edit Configurations > + (i.e. Add New Configuration) > Application
  2. Enter values as per below screen grab & save configuration.
  3. Next, click on Run > Run “Kafka”
  4. Intellij console will show log messages indicating that Kafka is running.
  5. Also, in terminal where Zookeeper is running, you will see logs indicating that Kafka has connected to Zookeeper.

七、测试 Kafka

  1. Create a new topic named “test” with a single partition and only one replica:
> bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

For Windows:

> bin\windows\kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
  1. We can now see that topic if we run the list topic command:
> bin/kafka-topics.sh --list --zookeeper localhost:2181
test

For Windows:

> bin\windows\kafka-topics.bat --list --zookeeper localhost:2181
test
  1. Run the producer and then type a few messages into the console to send to the server.
> bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
This is a message
This is another message

For Windows:

> bin\windows\kafka-console-producer.bat --broker-list localhost:9092 --topic test
This is a message
This is another message
  1. Kafka also has a command line consumer that will dump out messages to standard output.
> bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning
This is a message
This is another message

For Windows:

> bin\windows\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic test --from-beginning
This is a message
This is another message

参考链接