Writing

Elastic partitioning redefines Kafka partitions as movable units

Many people, myself included, treated Kafka's fixed partitions as a hard constraint for maintaining ordering and throughput. The familiar reasoning is that a partition is the unit of parallelism and ordering; if you need more throughput, add partitions; if you need order, put related keys in the same partition. That was long the sensible default. But it is not the whole story.

The article Can Kafka Support Elastic Partitioning? looks at elastic partitioning—dynamically splitting and merging key ranges. It argues that this can scale while preserving per-key ordering and batch efficiency. On the surface, that sounds like a way around the partition bottleneck. But the more I sit with the argument, the clearer it becomes that the real shift is not about removing partitions but redefining them as logical, movable units.

The real shift is not about removing partitions but redefining them as logical, movable units.

That distinction matters because it changes what client and broker software have to assume. Today, a partition is a stable home: a leader broker owns it, consumers read from a fixed sequence, and offset tracking maps cleanly to a partition ID. Once key ranges can split or merge, that stability disappears. Clients must track which broker owns which range at any given moment, and that tracking has to survive range moves.

First, consumer state locality breaks. In current Kafka idioms, a consumer instance often holds local state keyed to the partitions it is assigned. Rebalancing already has sharp edges; with movable ranges, the mapping between a consumer's local state and the ranges it is responsible for becomes much more fluid. Clients cannot simply say 'I own partition 3.' They have to say 'I own keys in this range, which currently lives on that broker.' That complicates both rebalancing logic and local state management.

Second, replay and recovery become harder. Offset tracking today is straightforward because a partition's offsets are a monotonic sequence within a stable log segment. With dynamic partitions, the log a consumer needs to replay may have split into two ranges, or two ranges may have merged into one. Offset tracking now spans the lifecycle of partitions: a checkpoint needs to know not just a number, but which range boundaries were in effect. Replaying from a previous state becomes an exercise in reconstructing partition history, not just seeking to an offset.

I do not read this as a case that Kafka cannot or should not support elastic partitioning. I read it as a warning about sequencing. If you adopt patterns like KIP-500 or tiered storage—moves that already shift assumptions about where state lives and how long it is retained—you will need new operational tooling for partition lifecycle management. That tooling has to handle range splits and merges, consumer reassignment across ranges, and offset reconstruction. It will be needed long before any of this becomes a default.

The article does not promise an easy path, which is why I keep returning to it. Kafka has always scaled by adding partitions. The harder problem is making that scaling feel continuous while preserving the ordering guarantees people rely on. That requires treating partitions less like fixed storage units and more like logical, movable ranges—and accepting that the operational layer has to catch up first.

Originally covered at terroir.systems ↗