Log files - srivalligade04/ConfluentExamPreparationNotes GitHub Wiki

What Are Kafka Log Files?

  • Kafka stores messages in log segments on disk. Each topic partition has its own directory and set of log files.
  • Each message written to Kafka is appended to a log file.
  • These logs are not logs in the traditional sense (like application logs), but rather the actual message data.

Kafka Log File Structure

  • Kafka stores logs in the log.dirs directory (configured in server.properties). The structure looks like this

image

Each partition directory contains:

  • .log: The actual message data.
  • .index: Maps message offsets to physical positions in the log.
  • .timeindex: Maps timestamps to offsets for time-based lookups.

Log Segmentation and Retention

Kafka splits logs into segments to manage size and retention:

Controlled by settings like:

  • log.segment.bytes (max size of a segment)
  • log.retention.hours / log.retention.bytes (how long or how much data to retain)
  • Old segments are deleted or compacted based on these policies.

image