The Phantom Reads 👻📚

Any system can experience phantom reads if it’s not designed carefully.

⭐️ phantom reads {AKA Ghost Reads}: the act of two identical read queries returning different messages.

The literature on the matter is mostly focused on databases in the context of Transaction Isolation, but the problem in Kafka is a matter of durability & consistency.

Imagine the following scenario:

1. Producer writes data with offsets 5-7

2. The leader broker (kafka-1) accepts it

3. The data is not yet replicated to the followers and the consumer reads data with offsets 5-7

   

4. The leader broker (kafka-1) dies. kafka-2 takes over.

5. Since neither kafka-2 or kafka-3 had the messages from 5-7 — the data is lost. 👋 The new log ends at offset 4.

6. The producer continues writing. It writes 5 new records - they’re now at offset 5-9.

7. The consumer reads data from offset 8 (its last read offset was 7). It misses the new records 5-7. Worse off - if the same consumer explicitly starts reading from offset 5 again, it will experience the ghost read. The data it receives for the messages at offset 5-7 will be different than the one it previously received.

How Kafka Avoids This

Fortunately - the case I presented to you is not reality. Kafka does not suffer from this problem, because the devs already took measures against it.

The High Watermark (HWM) ✨

The fix is incredibly simple - Kafka only returns fully-replicated messages to consumers.

In our example, the consumer read messages 5,6,7 before they were replicated. This cannot happen in practice.

The max readable offset is denoted by the high watermark offset. This is the highest offset up to which every in-sync replica has caught up to.

Follower brokers replicate by sending FetchRequests to the leader broker. The leader replies with the latest data and keeps a local map of follower broker → latest fetched offset per partition (!)^{Each partition has a different HWM offset.}

The lowest common denominator of the highest offset becomes the next high watermark offset (i.e the newest offset every follower has replicated).

The high watermark changes on more or less every follower→leader fetch request.

When the leader receives confirmation that every follower replica of the partition has read up to offset 7, the high watermark gets incremented to that. Only then is the consumer sent message 7.

HWM +1

The incrementation of the high watermark actually happens on the second fetch request after the one that replicates it.

In this way, it actually takes two rounds of replication fetch requests to be ready to expose tail data to consumers. Here’s how it works in detail:

The high watermark is actually exposed to consumers in each FetchResponse - that’s how consumer lag is calculated.

In other words, for all intents and purposes, the end of the partition’s log from a consumer’s point of view is the high watermark offset.

Simple solution! But it goes a long way. 🫡