How Kafka actually stores messages on disk

Kafka feels like a message queue, but under the hood it's much simpler: a set of append-only files on disk. Once you can picture those files, a lot of Kafka's behavior (why it's fast, how consumers replay old messages, what retention does) stops being mysterious.

This is the first post in a series on Kafka internals. We'll start with the storage layer.

Topic, partition, log

A topic is split into partitions. Each partition is an ordered, append-only log: new messages go on the end, and existing ones are never changed.

Every record in a partition has an offset. Offsets increase as records are appended, starting at 0, but they aren't guaranteed to stay consecutive: after log compaction (more on that below) some offsets are simply missing, so you can't assume offset 101 exists because 100 and 102 do. Offsets are per partition, so two partitions each have their own offset 0. Kafka only guarantees ordering within a partition, and by default messages with the same key go to the same partition (the key is hashed), which is how you keep related events in order.

What's on disk: one directory per partition

A topic called my-topic with two partitions becomes two directories on the broker, my-topic-0 and my-topic-1. Inside one:

my-topic-0/
  00000000000000000000.log
  00000000000000000000.index
  00000000000000000000.timeindex
  00000000000000368769.log
  00000000000000368769.index
  00000000000000368769.timeindex

The log isn't one giant file. It's cut into segments, and each segment is a group of files that share a name: the offset of the first message in that segment, padded to 20 digits. The second group above starts at offset 368,769.

• .log holds the messages themselves.
• .index helps find a message by offset.
• .timeindex helps find a message by timestamp.

(There are a few small bookkeeping files too.) Only the newest segment, the active segment, is ever written to. Older ones are read-only.

How a segment rolls

A new segment typically starts when the active one reaches its size or age limit. The defaults are segment.bytes = 1 GiB and segment.ms = 7 days (the broker-wide versions are log.segment.bytes and log.roll.hours), and whichever comes first wins. Other conditions can also roll a segment, such as an index file filling up. Segments are what make the rest of Kafka's design work, as you'll see in a moment.

What's inside a segment

Producers send records in batches, and Kafka stores them as batches, in the same format that goes over the network. The docs highlight why: the broker can pass a batch from producer to disk to consumer without recopying or converting it.

A .log file is just a series of these batches:

00000000000000000000.log
  Record batch
    base offset, record count
    timestamps
    compression setting
    the records themselves
  Record batch
    ...

Each batch carries its own metadata, so records can be compressed together as one unit and the broker can read a batch's offsets and timestamps without unpacking every record.

Reading by offset: find the segment, then the position

Say a consumer asks for offset 368,800.

1. Pick the segment. Kafka keeps the segments' starting offsets in memory and searches them. The right one is the segment with the largest starting offset that isn't higher than the one you want. Here that's 00000000000000368769.
2. Use the index. The .index file maps offsets to byte positions in the .log file. It's sparse: by default it adds an entry roughly every 4 KiB of data (index.interval.bytes), not one per record. So Kafka finds the nearest entry at or before the target.
3. Scan forward. From that byte position, read through the record batches until the requested offset.

Because the index is sparse, it stays relatively small. The scan is usually short, but how short depends on the index spacing and the size of your batches, since Kafka reads whole batches, not individual bytes.

The .timeindex helps with "give me messages from 3 pm". It maps timestamps to offsets, but it's an approximate starting point: Kafka uses it to jump close, then checks records to find the first one at or after your timestamp.

You can look inside a segment yourself with the tool Kafka ships:

bin/kafka-dump-log.sh --files /path/to/my-topic-0/00000000000000000000.log

Why it's fast: the page cache

Kafka doesn't keep its own big in-memory cache of messages. The docs say it relies heavily on the filesystem for storing and caching messages.

On a write, Kafka appends to the log, which really means the data lands in the operating system's page cache first. The OS writes it to disk later. Writes are sequential appends, which disks and SSDs are good at, and as the docs note, reads don't block writes or each other.

On a read, a consumer that's keeping up asks for data that was just written, so it's usually still in the page cache. The docs say you'll see no read activity on the disks at all in that case. Kafka then uses sendfile to hand the data from the cache to the network socket without copying it through the application. (One exception: with SSL/TLS the data has to be encrypted in user space, so sendfile isn't used.)

A consumer that's far behind, such as one replaying last week's data, reads from disk instead, which is slower and can push recent data out of the cache.

Is data really safe if it's only in the page cache?

This surprises people. By default Kafka doesn't call fsync on every write. Flushing is left to the OS, and the flush settings default to effectively "never force it". The docs say plainly that Kafka doesn't want to require fsync on every write for its guarantees.

Kafka relies on replication instead. Each partition is copied to other brokers, and the docs say a committed message won't be lost as long as at least one in-sync replica is alive. That promise has conditions, though. It depends on your settings:

• A replication factor above 1. One copy means no redundancy.
• acks=all on the producer, so the write waits for the in-sync replicas. (It's the default in current versions.)
• min.insync.replicas set high enough. It defaults to 1, and with acks=all that means a write can succeed with only the leader holding it. Setting it to 2 with a replication factor of 3 is a common choice.
• Unclean leader election disabled. It's off by default. If enabled, an out-of-sync replica can become leader, and records it never received are lost.

Even then, replication doesn't cover everything. If all the replicas lose their unflushed data at once, for example in a data-center-wide power cut, records can still be lost.

This is a different trade-off from the database approach in my post on how transactions and WAL work, where one machine flushes its log on commit. We'll cover replication in a later post.

Retention: deleting whole segments

Under the default delete policy, Kafka removes whole log segments, not individual records. By default it keeps data for 168 hours (log.retention.hours), with no size limit (log.retention.bytes is -1), and checks every five minutes. When a segment is old enough, the whole segment is removed, which is cheap: it's just deleting files. (The active segment isn't deleted until it rolls.)

Log compaction (cleanup.policy=compact) is different, and it does remove individual records. Instead of deleting by age, Kafka keeps at least the latest record for each key, and a background process rewrites older segments to drop the stale ones. Surviving records keep their original offsets, which is why compacted logs have gaps. It suits topics that represent "current state per key".

What to take away

• A partition is an ordered, append-only log, stored as segments, and each segment is a .log file plus two small index files.
• Segment names are the first offset they contain.
• Reads find the segment, use a sparse index, then scan forward through batches.
• Speed comes from sequential I/O, the page cache and zero-copy reads.
• Durability comes from replication, and only with the right settings (acks=all, a sensible min.insync.replicas, no unclean leader election).
• Delete-based retention removes whole segments. Compaction rewrites them.

If you run Kafka, one practical point follows: a lot of its performance depends on the page cache, so leave plenty of free memory for the OS and don't hand it all to the JVM.

References: the Apache Kafka documentation, especially the Design and Log implementation pages and the broker configuration and topic configuration references.