In 2025, IBM joined the chat and the Diskless architecture went open-source.

Top News

• Snowflake almost acquired RedPanda (Jan 2025)

  • 💀 This revealed RedPanda has just ~$20M in ARR⁴ …

  • RedPanda pivots massively to AI (Oct 2025).

• 🏆 Aiven announces Diskless Topics KIP (May 2025)²  

• ‼️ IBM buys Confluent for $11B (Dec 2025)

Others

• Databricks made some moves into streaming:

  • Announced Zerobus, a Kafka-like message bus for ingesting data directly into your Lakehouse in a Unity Table format

  • Spark Real-Time Structured Streaming released, a new Spark Structured Streaming feature (named awfully) that processes events in tens of milliseconds

  • _{hell, they even released a paper bashing Kafka}

• Snowflake didn’t wait long either - they announced Snowpipe, a real-time ingestion method allowing apps to write rows directly into Snowflake tables

• Cloudflare released Pipelines and the “Cloudflare Data Plaform”, a way to receive events via HTTP, stream-process transform them with SQL and ingest into R2 (their object store) in an Iceberg format)

• S2 launched officially

We seem to be seeing every vendor nibbling at the edges of real-time streaming.

Kafka Ecosystem 😇

• Kafka 4.1 released

• Kafka 4.0 released 🎉

  • KIP-848 (the new consumer group protocol) went GA

  • KIP-932: Queues - went EA

  • ZooKeeper was officially completely removed

Top Trends

• 🧊 Iceberg and the Lakehouse is everywhere

  • Confluent Cloud Tableflow went GA (for Iceberg)

  • WarpStream released their own Tableflow

  • Aiven released (open-source) Iceberg Topics

  • I wrote how I think Iceberg will eat up some of Kafka’s single source of truth use case. I see the industry speed-running into tihs.

• 💾 S3 is eating the (Kafka) world

  • S3 Express lowered their prices 30-85%, making it significantly more cost-effective to build Diskless Kafka on top of S3 Express

  • Two KIPs were proposed to have efficient reads of cold data from object storage (KIP-1255: Remote Read Replicas and KIP-1248: Consumers reading from Object Storage)

  • As of writing, I count 10 diskless direct-to-S3 Kafka implementations - IBM Confluent Freight, IBM Confluent WarpStream, AutoMQ, Bufstream, Tansu, Aiven Inkless, (soon-to-be) Kafka Diskless Topics, StreamNative Ursa, Kafscale, RedPanda Cloud Topics

• 🤖 Everyone is building “AI Agent infra”

  • Confluent releases Streaming Agents and Confluent Intelligence (“Real-Time AI with Kafka”)¹

  • Streamnative releases an Orca Agent Engine

  • RedPanda’s whole pivot into an Agentic Data Plane

  • Stan: “the last thing AI agents need is Kafka…”

• 💸 Everyone was racing to claim the lowest costs:⁵

  • StreamNative: “Cut Kafka Costs by 95%” / “How We Run a 5 GB/s Kafka Workload for $50/h”

  • Aiven: “Diskless Kafka: 80% Leaner, 100% Open”

  • Confluent:

    • “Confluent. Will. Never. Lose. On. Price”

    • “Why Hosted Apache Kafka Leaves You Holding The Bag”

    • “Confluent Freight are up to 90% cheaper than self-managing Kafka”

  • Buf: “Bufstream: Kafka at 8x lower cost”

  • Remember, if everybody is claiming to be cheaper - nobody is. Last year I the lies vendors make: “The Brutal Truth About Kafka Cost Calculators”

  • That being said, some changes happened:

    • Confluent lowered their retail prices, especially of Freight.

    • Aiven released a generous free tier (Dec, 2025)

Parting Words

If I’ll remember 2025 by anything, it’ll be by its AI slop:

• “The Future of AI Agents is Event-Driven” - Confluent

• “AI Agents at Scale - The Critical Role of Kafka and Flink” - Conduktor

• “The Future of Data Streaming with Apache Flink For Agentic AI”

• Building a AI Agent Meal Planner with MongoDB, Kafka and Flink - Confluent (WTF?)

  

  The most ridiculous thing I’ve seen in 2025

No seriously, nothing is more telling of the fall of Data Streaming when its vendors need to create these obscure overly-complex buzzword-driven architectures³ for an application that… processes at most a few requests per second.

The Meme of 2025

More Writing? 🔥

Here is a short selection of some of my best-performing pieces of content from 2025.

I made the Hacker News front page 4 times this year. The internet found them very interesting, so I assume you will too!

• MongoBleed explained simply

• Kafka is fast — I’ll use Postgres

• Why Was Apache Kafka Created?

• how AWS S3 serves 1 petabyte per second on top of slow HDDs

1  The only thing missing is a Blockchain

2  There was some contention on whether this should be the path Kafka takes, as it prompted Slack to publish KIP-1176 which moved Kafka in the same direction but was as diskless. Great progress was made as on Dec 15 2025, Slack announced their intention to withdraw KIP-1176 in fear of not fragmenting the community, and announced their intention to help contribute to KIP-1150.

3  In a way, I fear this is the future of the internet. AI generated articles talking about AI vaporware, formatted carefully to dish out literally every buzzword out there with the hopes of winning some SEO ranking.

4  While this number is very small, one important reminder that people miss out is RedPanda’s type of revenue. Since they sell BYOC, this number tends to be pure profit (excluding R&D and etc.). It is therefore unfair to compare it to SaaS products, because those resell AWS infrastructure which reduces profits but bumps up the revenue. If you want to compare both, I would apply a 2-3x multiple on BYOC revenue.

5  No wonder people call it a race to the bottom. “Is it a race to the bottom for streaming infrastructure pricing?” (Reddit)