Kafka - keshavbaweja-git/guides GitHub Wiki
Introduction
Kafka is a
- distributed
- scalable
- replicated
- partitioned
- fault tolerant
- stream processing platform.
Key Concepts
- Kafka runs on a distributed cluster of servers.
- Records are published to and stored in distributed topics.
- Each record consists of a Key, Value and Timestamp.
Topics and Logs
- Topic is a category to structure message publication
- Topic is always multi subscriber
- Topic is implemented as partitioned, ordered, immutable append only commit log.
- Topic partitions allow a topic to expand beyond storage capacity available on one node.
- Each record in a partition is assigned a sequential id number called the offset.
- All published records are durably persisted for a configurable retention period.
- Only metadata retained on a per-consumer basis is the offset of the consumer in the partition log.
- Each partition has one server that acts as "leader" and zero or more servers that act as "followers".
- The leader handles all read-write requests for the partition while the followers passively replicate the leader.
Security
// 1. Generate server keystore (stores server identity)
keytool -keystore kafka.server.keystore.jks -alias localhost -keyalg RSA -validity {validity} -genkey
// 2. Generate CA certificate and key
openssl req -new -x509 -keyout ca-key -out ca-cert -days {validity}
// 3. Generate client truststore importing CA certificate created in step 2
keytool -keystore kafka.client.truststore.jks -alias CARoot -import -file ca-cert
// 4. Generate server truststore importing CA certificate created in step 2
keytool -keystore kafka.server.truststore.jks -alias CARoot -importcert -file ca-cert
// 5. Export server certificate from server keystore
keytool -keystore kafka.server.keystore.jks -alias localhost -certreq -file cert-file
// 6. Sign server certificate with CA certificate
openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out cert-signed -days {validity} -CAcreateserial -passin pass:{ca-password}
// 7. Import CA certificate in server keystore
keytool -keystore kafka.server.keystore.jks -alias CARoot -import -file ca-cert
// 8. Import server certificate in server keystore
keytool -keystore kafka.server.keystore.jks -alias localhost -import -file cert-signed