Diskless Kafka: What Happens When Brokers Stop Owning the Data?

Apache Kafka was designed to make good use of sequential I/O. Producers batch records, brokers append those batches to ordered log segments, and consumers read sequential ranges. Kafka also relies on the operating system's page cache and uses zero-copy techniques such as sendfile where possible. The Kafka design documentation explains why this works: batching turns many small operations into larger sequential reads and writes, which storage devices and operating systems can handle efficiently. In the cloud, another storage option is available. An object store can provide durable, replicated storage independently of broker compute. Could Kafka use that service for the active log and let brokers scale without moving the durable dataset? That is the idea behind diskless Kafka : retain Kafka's client API while moving durable payload storage away from broker-owned partition logs. The implementation still has to preserve ordering, assign offsets, track committed writes, and serve reads efficiently. Different systems divide these responsibilities differently, and their feature coverage varies. Upstream status, August 25, 2026: KIP-1150 is accepted, but native Diskless Topics are not yet a generally available Apache Kafka feature. KIP-1163 and KIP-1164, which define the core implementation, remain under discussion. How the broker and the log are coupled In classic Kafka, a partition leader accepts records, assigns offsets, and appends batches to local log segments. Followers fetch from the leader and append the same batches to their own storage. With acks=all , the producer waits for the required replication before receiving a successful acknowledgement. Batching reduces network and system-call overhead, while the page cache can serve recent data without a physical disk read. A broker therefore owns both the processing responsibility for its partitions and their local log segments and replica state. Changing where partitions run often means moving their data too. The cloud changed the storage equation Consider a cluster spread across three availability zones. Each record is stored by the leader and copied to followers in the other zones: Replicating across zones lets Kafka tolerate zone failures when replica placement and acknowledgement settings are configured accordingly. It can also generate substantial cross-AZ traffic. KIP-1150 identifies this as a major cost driver on AWS and Google Cloud. The economics depend on provider and network configuration; payload replication is only one part of the total traffic bill. Every broker also needs enough storage for its assigned partitions. Adding capacity can require partition reassignment; replacing a lost replica requires copying its data. A broker holding terabytes of partition state takes longer to replace than a worker whose durable state lives elsewhere. Storage capacity and data movement constrain how quickly compute can change. Amazon S3, Google Cloud Storage, and Azure Blob Storage provide durable storage independently of Kafka brokers. With an appropriate regional storage class, the provider also handles replication across failure domains. The useful comparison is between that service and Kafka's complete replicated storage path. EBS itself replicates within one availability zone ; Kafka's replicas can extend protection across zones. KIP-1150 proposes delegating payload durability to object storage to...