Writing

The MiniMax M3 optimization lesson: chasing the moving bottleneck

I keep coming back to the vLLM team's MiniMax M3 optimization on AMD's MI355X because it is a masterclass in not stopping at the first win. The published result was a 3.14x throughput improvement, which is easy to read as one big jump. The more instructive part is the sequence of checks the team performed, because each one removed a separate source of waste that was not obvious at the start.

• Check the real tensor shapes that land on each rank after tensor-parallel sharding.
• Fuse the shared expert into the same grouped GEMMs as the routed experts instead of running a redundant layer pass.
• Confirm that the claimed fast paths actually executed; assumption is not instrumentation.
• Reuse sparse-attention block selections across adjacent layers instead of recomputing per layer.

The first check is the least glamorous but probably the most important. It is easy to assume that tensor-parallel sharding divides the work evenly, but the real shapes that land on each rank can differ because of padding, uneven sequence distribution, or a mismatch between the model config and what the runtime actually does. Checking those shapes before doing any kernel work removes a whole class of mental models that are wrong in ways that only show up later as unexplained load imbalance.

The second check found work that was repeated every layer. MiniMax M3 has a shared expert that runs regardless of routing, and in the initial implementation it was launched separately from the routed experts. That separate launch added kernel dispatch and memory movement costs across every layer. Fusing the shared expert into the same grouped GEMMs as the routed experts removed what amounted to a redundant layer pass. At low concurrency, where those overheads are a larger fraction of total latency, this fusion gave up to 30%.

The third check is about not confusing a capability with a result. A kernel can have multiple implementations, and seeing a kernel name in a trace does not tell you which code path actually executed. The team had to confirm that the claimed fast paths were being hit, not just that the kernel was present. The phrase from the post is worth remembering: assumption is not instrumentation. This is a common failure in performance work: the optimized path exists, but a config flag or a shape mismatch silently falls back to a slower path.

The fourth check moved from compute to attention. Sparse attention block selections were being recomputed for every layer, even when adjacent layers could reuse the same selection. Recomputing per layer adds redundant CPU work and scheduling overhead. Reusing the block selections across adjacent layers removes that overhead and is especially relevant when the model's sparse patterns are relatively stable across depth.

The 3.14x throughput jump came from chasing the bottleneck as it moved. After the first fix, the limit shifted from compute kernels to memory bandwidth; after the next, it shifted again to queueing and launch overhead. That is not a sign that the earlier fixes were wrong. It is what happens when you keep measuring instead of declaring victory after the first win. Each step revealed the next most expensive thing.

For enterprise inference tuning, the useful next-step filter is to ask what changed in the profile, not just what improved. If throughput went up, the next bottleneck is now more visible. If it did not improve as expected, check whether the optimized path actually ran. The vLLM team's MiniMax M3 work on MI355X is a useful case study because it shows how a large aggregate gain can be made of several smaller, verifiable steps, each of which required looking at the system as it actually behaved.

Originally covered at vllm.ai ↗