Writing

The hard part isn't the model choice; it's the harness

Most conversations about building agentic AI systems start with model selection. I understand why. Frontier models are improving quickly, and choosing the right one feels like a strategic decision. But when I look at what actually breaks in production, the model choice is rarely the hard part. The hard part is the harness around it.

Microsoft's Copilot harness in Foundry makes that explicit. It routes work across 20+ frontier models, manages sandboxing and governance, and now runs on Rust for performance. On the surface, that reads like an infrastructure detail. In practice, it is the part of the system that determines whether agentic AI can run safely and reliably at enterprise scale. The harness is not a wrapper around a model. It is the control plane that sits between the model and everything the model is allowed to touch.

The reason this matters is straightforward. Agent loops fail in the control plane, not in the model call. A single model call is a request and a response. An agent loop is a sequence of decisions, tool calls, retries, context updates, and policy checks. The model is only one component in that sequence. When something breaks, it usually breaks in the orchestration around the model: a bad tool result gets fed back into the context, a permission check is missed, a sandbox boundary is too loose, or a retry loop spins out of control. None of those failures are solved by switching to a better model.

Managed orchestration reduces that operational risk. Instead of every team building its own ad hoc control plane, they get a shared layer that handles routing, sandboxing, and governance consistently. That consistency is what makes GenAI viable inside a large enterprise. It also matters because model access is getting commoditized. It is already easy to swap one frontier model for another. What is not commoditized is the layer that decides which model to use, what it is allowed to do, what it can access, and how its outputs are audited. The difference is governance.

Governance is not a checkbox. It is the set of rules that determine whether an agent can act on a user's behalf, which systems it can reach, what data it can read, and how every action is logged for audit. Without that layer, models are powerful but unusable in a regulated environment. With it, an organization can adopt new models as they improve without rebuilding its safety and compliance posture from scratch.

Control planes age better than prompts.

That is the lesson I keep coming back to. Prompts are tightly coupled to a specific model and a specific version. A control plane, if designed well, persists across model changes. It encodes the rules that don't change when the underlying model improves: identity, permissions, data boundaries, logging, retry policy, fallback behavior. Those things outlive any single model generation. Investing in the harness is not as exciting as picking a model, but it is the investment that compounds.

When teams ask me where to focus, I don't point them at model benchmarks. I point them at the harness. The model choice will keep changing. The control plane is what stays.

Originally covered at blogs.microsoft.com ↗