Writing

The Real Bottleneck in Data Platforms Is Neglected Orchestration

Breaking up application monoliths into microservices has become a standard move for organizations looking to ship faster and scale independently. But while the software world has largely internalized this pattern, the data world is just beginning its own version of the journey. The monoliths here are the all-in-one data platforms that bundled storage, transformation, and orchestration into a single stack. Teams are increasingly drawn to best-of-breed tools for each layer, yet the conversation often stays stuck on tool sprawl. I think that focus misses the real bottleneck.

What gets overlooked is the orchestration layer. Too many teams treat it as an afterthought, wiring together tools with a basic scheduler and calling it a day. They treat orchestration like a cron job: fire this task at 3 a.m., email on failure. That approach breaks down as pipelines grow in complexity and as the business comes to depend on fresh, accurate data. Orchestration isn't just a trigger mechanism. It's the control plane for your data platform, responsible for coordination, dependency management, and recovery from partial failures.

A control plane demands clear service-level objectives. Data freshness — how current the data needs to be for consumers — isn't a nice-to-have; it's a contract. Recovery from failures shouldn't be a manual scramble at 6 a.m. because nobody defined what 'acceptable recovery time' means. And cost is easy to ignore when orchestration doesn't let you reason about how resources are provisioned and released. Without SLOs around freshness, failure recovery, and cost, the orchestration layer is blind, and teams end up firefighting instead of improving.

The move toward best-of-breed tools amplifies this. When you decouple storage, transformation, and serving layers, the coordination burden shifts squarely onto orchestration. A tightly integrated stack might have handled state internally; now you have to manage it explicitly across separate systems. That can actually be a good thing if you invest in the orchestration layer up front. But too often, teams adopt specialized tools without strengthening the glue that holds them together, and then wonder why reliability erodes.

There are still cases where a tightly integrated stack makes sense. When data volumes are small and the pipeline is simple, an all-in-one platform can reduce operational overhead. You don't need sophisticated orchestration to keep things running. But that window closes as soon as the data footprint grows or the business starts asking harder questions. At that point, the lack of orchestration SLOs goes from a non-issue to the primary source of grief.

The article goes into more detail on why this pattern is repeating across the industry and what it looks like when orchestration SLOs finally catch up with a team. The bottom line is that the data platform's control plane deserves the same rigor we apply to the services we build. Otherwise, the migration away from monoliths just swaps one kind of fragility for another.

Orchestration isn't a cron job with a dashboard. It's the system that decides whether your data arrives on time, whether failures cascade, and whether you're burning money on idle compute. Treating it as anything less is a bet that your data pipelines will stay simple forever. That's rarely a bet worth taking.

Originally covered at dataopsleadership.substack.com ↗