Writing

Why I'd instrument before building an internal AI cluster

Nutanix built a $20 million internal AI cluster to reduce its reliance on SaaS AI tools like Copilot and Claude, expecting to break even in about a year. That's a significant move, but at enterprise scale it's not hard to see the appeal. When you're already spending heavily on SaaS AI, the per-token costs add up fast. Owning the capacity changes that math.

The case for owning the hardware is straightforward. You get a predictable cost per request instead of a bill that scales with usage. You can lower latency because traffic stays on your network. You get full control over data, which matters for security and compliance. If Nutanix truly breaks even in about a year, they're not buying hype. They're buying a pricing floor for heavy internal use.

But there's a real case against it. Hardware ages, and models improve quickly. A cluster optimized for today's open-weight models may not run tomorrow's frontier workloads well. You also inherit all the operational burden: MLOps, security patching, capacity planning, and utilization risk. Idle GPUs are expensive. SaaS vendors handle that burden for you. Utilization is the whole game.

I wouldn't frame this as a simple build versus buy decision. I'd start by instrumenting actual inference volume, latency, and failure modes. That gives you a baseline before you spend anything. You need to know which workloads are heavy, which are latency-sensitive, and which fail in ways that matter. Without that data, you're guessing.

Once you have the data, I'd split the workloads. Self-host the high-volume, deterministic tasks like code completion or internal summarization on smaller models that run efficiently on your own hardware. Keep SaaS for complex reasoning or low-volume, high-stakes work where vendor models are worth the premium. Add a routing layer to send each request to the right backend, and cache aggressively to avoid repeat cost.

The metric that matters is cost per successful request, not raw token price. A cheaper model that fails more often can be more expensive overall. Re-run those numbers monthly because models, volumes, and prices change. Nutanix's one-year ROI is impressive, but it only holds if utilization stays above the break-even point. That's the metric to watch. If your own cluster sits idle, you've built a very expensive paperweight.

Originally covered at theregister.com ↗