[Docker] kafka & replica set - ysp-laonpeople/DOC GitHub Wiki

docker를 이용한 kafka

  • docker를 사용해 kafka를 실행합니다.

docker-compose.yml

version: "2"
services:
  zookeeper:
    image: "confluentinc/cp-zookeeper:latest"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    ports:
      - "22181:2181"
    volumes:
      - "d:/docker/zookeeper/data:/var/lib/zookeeper/data"
      - "d:/docker/zookeeper/log:/var/lib/zookeeper/log"
  kafka:
    image: "confluentinc/cp-kafka:latest"
    depends_on:
      - "zookeeper"
    ports:
      - "9092:9092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181"
      KAFKA_ADVERTISED_LISTENERS: "PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092"
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: "PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT"
      KAFKA_INTER_BROKER_LISTENER_NAME: "PLAINTEXT"
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
    volumes:
      - "d:/docker/kafka/data:/var/lib/kafka/data"

volumes

  • container가 삭제되거나 재기동시 topic과 offset정보 유지를 위해 절대경로를 마운트합니다.
    volumes:
      - "d:/docker/kafka/data:/var/lib/kafka/data"

실행

  • docker-compose.yml 파일의 폴더로 이동합니다.
  • 다음 명령어로 실행합니다.
docker-compose up -d

replica set

  • 장애 상황 대응을 위해 replica set으로 묶어서 관리가 가능합니다.

docker-compose.yml

version: '2'
services:
  zookeeper-1:
    image: confluentinc/cp-zookeeper:latest
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    ports:
      - 22181:2181
    volumes:
      - "d:/docker/zookeeper/data1:/var/lib/zookeeper/data"
      - "d:/docker/zookeeper/log1:/var/lib/zookeeper/log"      

  zookeeper-2:
    image: confluentinc/cp-zookeeper:latest
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    ports:
      - 32181:2181
    volumes:
      - "d:/docker/zookeeper/data2:/var/lib/zookeeper/data"
      - "d:/docker/zookeeper/log2:/var/lib/zookeeper/log"      
  
  kafka-1:
    image: confluentinc/cp-kafka:latest
    depends_on:
      - zookeeper-1
      - zookeeper-2

    ports:
      - 29092:29092
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper-1:2181,zookeeper-2:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka-1:9092,PLAINTEXT_HOST://localhost:29092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
    volumes:
      - "d:/docker/kafka/data1:/var/lib/kafka/data"      
  kafka-2:
    image: confluentinc/cp-kafka:latest
    depends_on:
      - zookeeper-1
      - zookeeper-2
    ports:
      - 39092:39092
    environment:
      KAFKA_BROKER_ID: 2
      KAFKA_ZOOKEEPER_CONNECT: zookeeper-1:2181,zookeeper-2:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka-2:9092,PLAINTEXT_HOST://localhost:39092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
    volumes:
      - "d:/docker/kafka/data2:/var/lib/kafka/data"
  • 2개의 kafka와 실행을 위한 zookeeper 2개가 필요합니다.
    • 향후 kfaka실행에 zookeeper가 필요하지 않을 수도 있다고 합니다.

environment

  • broker id를 각각 설정합니다.
    environment:
      KAFKA_BROKER_ID: 2

실행

  • 실행시 zookeeper가 알아서 leader와 follower를 설정합니다.
docker-compose up -d

테스트

  • broker id 1, 2를 각각 종료 후 테스트합니다.
  • topic이 정상적으로 조회됩니다.
  • image