As we covered last time - open source activity is absolutely buzzing.

Since then, it’s only been getting louder. April recorded the highest number of KIPs ever submitted to Kafka in a month, and among them was KIP-1150: Diskless Topics! 🏆

This is a pretty large KIP, driven by Aiven. It’s not yet accepted, but it is seeing heavy investment, is well thought-out and is the next logical step for Kafka. It’s very likely it’ll get accepted in one way or another.

That being said - it’s still WIP in the DISCUSS stage. Things mentioned here are subject to change.

What is Diskless?

In case you’ve been living under a rock, the Kafka space has been on a crazy cost-deflation and infra-simplification frenzy for the last two years.

It all came from these simple realizations:

The result? Your storage costs ~20x less, your cross-zone data transfer costs ~infinitely less (they’re zero).

Diskless 101

This isn’t a new cluster type like WarpStream - it’s simply a new type of topic in Apache Kafka. Diskless topics is the name.

When enabled (via diskless.enable), the topics will:

• be leaderless - any broker can serve produce/fetch requests for their partitions

• write directly to S3 - no replication for diskless topics between brokers

  • (I say S3 for clarity, but it works with any cloud object store like GCS or Azure’s)

• store a SharedLogSegment file in S3 that consists of many different partitions’ data from that time period.

  • (unlike today where a segment has only one partition’s data)

• scale up/down instantly and easily - when you start up new brokers, clients writing to diskless topics will naturally and automatically rebalance to them.

Write Path 101

• producers bootstrap and receive a (modified) metadata response containing the one leader broker they should write to.

• brokers accumulate data from producers for either 250ms or 8 MiB (whichever comes first; configurable), then persist it in a SharedLogSegment (SLS) to S3.

• The broker first writes the SLS to S3, then sends a request to the BatchCoordinator to commit the offsets for the data.

  • at the time of writing to S3, the data does NOT have offsets.💡

• The BatchCoordinator acts as a sequencer and persists the offsets in its backing store. It essentially stores the coordinates for an offset - “which file in S3 contains this record?”

  • It’s a pluggable interface and the plan is to have a default implementation backed by a topic called __diskless-metadata and using SQLite as a way to materialize it and enable efficient queries on it.

  • Aiven currently have a working implementation using just Postgres.

• The write-capable coordinator is the leader of the __diskless-metadata partition. The replicas of the partition are read-capable coordinators.

• a distributed cache is used so the reads are efficient

  • each AZ has a copy of the cache of the segments

  • it uses the Infinispan distributed cache library

  • on writes, the cache is also populated. This means a broker writes simultaneously to the cache and to S3

• the file stored in S3 is a SharedLogSegment, consisting of many different partitions’ data from that time period.

  • (unlike today where a segment has only one partition’s data)

• periodic merging of the small SLS files into larger ones will be done (KIP is still very unclear)

For more on the architecture, see Aiven’s docs.

Inkless vs Diskless

Disklesss is the open source KIP-1150.
Inkless is Aiven’s MVP implementation, which they also open sourced (under APGL).

There are differences - Diskless is not yet fully designed/implemented, and Inkless is.

• Inkless uses Postgres as the batch coordinator, directly utilizing Postgres functions.

• Inkless uses the Distributed Infinispan Cache; Diskless doesn’t even have a KIP for this.

Aiven is not accepting patches in the Inkless repo - their goal is to delete the repo once the KIP gets merged. But they want to work in public (which is really cool)

Most interesting in that repo is their commit history. It seems like they started work on this ~8 months ago, with a team of just 4 people. That’s realy impressive progress.

🤨 What’s the Catch?

Latency.

Latency suffers massively in this Diskless architecture.

The Inkless repo explains the latency breakdown well:

• Writes consist of:

  • commit interval - up to 250ms by default (inkless.produce.commit.interval.ms)

  • S3 PUT latency - 200-400ms p99 latencies for 2-8MB segments

  • Batch Coordinator commit - 10-20ms p99 for the Postgres implementation

• Reads consist of:

  • Finding the (S3) record-batch coordinates for the given offsets

    • ~10ms p99

  • Planning the fetching strategy (what comes from cache, what from S3)

  • Fetching from S3 and Cache

    • p99 of ~100-200ms for 2-8MB objects

Given this data, you’d expect a p99 latency of at most ~670ms for writes and ~250ms for reads. An end to end latency of no more than 1000ms.

Despite that, the doc in the Inkless repo shows higher e2e latency of up to 2.4 seconds - perhaps pointing towards opportunity for optimization.

The Annals of Diskless

Diskless’ brief history shows how WarpStream forever changed the Kafka space with their launch.

• August 2023: WarpStream launched

• May 2024: Confluent Freight announced

• May/July 2024: StreamNative Ursa, Bufstream, AutoMQ S3 launched

• Sep 2024: Confluent buys WarpStream for $220M; RedPanda announced intention for Cloud Topics

• April 2025: KIP-1150 was published by Aiven

• May 2025: Aiven releases the code for KIP-1150 (under AGPL) and offers it in their BYOC product.

Why Does Diskless Matter?

1. ⭐️ Open Source - because it’s open source, you are not locked in to any vendor. Also - it allows you to actually pocket the cost savings.

2. 💸 Cost - Diskless Kafka will cost you 11.2x less in infra fees than e.g. Confluent Freight ($100k/yr vs $2M/yr)

   1. Note that in Azure (where inter-zone networking is free) Diskless does not materially reduce the bill.

   2. Also note that under a certain threshold, the Diskless approach is not cheaper (because of the S3 PUT overhead)

3. 🌞 Simplicity - because it’s stateless, you eliminate a whole host of issues (IOPS, disk space, rebalancing partitions)

4. ✨ Flexibility - Kafka will be the only system in the market to support both slow and fast topics in one cluster.

🍱 Single Takeaway

🎬 Fin.

That’s it for this edition. Here is what else I can offer:

• ⭐️ Diskless Solutions Comparison (coming soon) - I will publish a long-form breakdown and comparison of every diskless solution out there. Subscribe to my long-form newsletter to not miss that - ✅ https://bigdata.2minutestreaming.com/.

• ⭐️ Technical Details about Diskless (2:20hr podcast) - there is no better technical deep dive into KIP-1150 out there on the internet. Make sure to watch it if you’re interested 👇

LIKED THIS?

If you find this content valuable, subscribe for more:

More Kafka? 🔥

Here’s what we posted on social media most recently:

OpenAI’s Kafka Architecture

What Queue do you choose

(this created some great discussions on hackernews too)

Low Latency Use Cases

Batch Size vs Linger Ms

Kafka’s Schema Problem

4 Ways KIP-405 Serves Reads Fast 🔥