Writing

Boring Infrastructure Sleeps Better

I keep coming back to a simple observation: the newest stack often adds operational risk before it removes any. It is easy to assume that a painful system needs a modern replacement, but the replacement brings its own unknowns. Unfamiliar failure modes, a smaller pool of operators, and less battle-tested behavior all carry a cost. In many cases, you are trading a known annoyance for an unknown outage.

Real resilience gains can come from deliberately unglamorous moves. The linked piece from Rogue Amoeba describes a concrete example. They swapped CGI for FastCGI and MySQL for SQLite on primary orders. Those are not exciting technologies by current standards. The results, though, matched what boring infrastructure often provides: performance improved, backups got simpler, and operational complexity dropped. Nothing about that list is flashy, but each item directly reduces the day-to-day burden of keeping a system healthy.

Boring infrastructure tends to sleep better. That phrase from the article captures the point: a smaller, more predictable stack means fewer moving parts to wake you at 3 a.m. FastCGI may be old, but it removes a process-spawning overhead that CGI carries. SQLite may not be the default choice for a busy service, but for the right workload, it can replace a database server with a file that is easy to back up and restore. The gain is not novelty; it is a clear reduction in operational surface area.

Not every legacy system is safe, though. Some are boring and brittle. A stack can be stable and still be one bad reboot away from a long night. Age alone does not make something resilient; a system can be old, poorly understood, and running on unpatched dependencies. The lesson is not to preserve old technology for its own sake. The lesson is to choose the simplest upgrade that reduces the pain you actually have, rather than the one that feels most modern.

The practical takeaway is to measure current pain first. Before deciding what to replace, understand where the system actually hurts: slow requests, complex backups, noisy alerts, or operational toil. That points to a specific bottleneck. Then pick the simplest upgrade that reduces that pain. In Rogue Amoeba's case, the pain told them to move off CGI and MySQL, not to adopt a distributed database or rewrite in a new language. The upgrade was deliberately modest, and that is what made it effective.

I try to apply that standard when I evaluate infrastructure choices. If a change does not reduce an observed operational problem, it is probably premature. If it adds new components, new failure modes, or new operational unknowns, the burden of proof should be high. The most resilient systems are often built on tools that are no longer fashionable. They are not exciting, but they are understood. And that understanding, more than any new feature, is what lets the people responsible for them sleep.

Originally covered at weblog.rogueamoeba.com ↗