Writing

The availability risks that never go away

Most post-incident reviews I've read end up pointing at something specific: a bad config, a missed limit, a human error. The underlying assumption is that if process were thorough enough, those failures could be prevented. It's a comforting idea because it suggests reliability is a matter of finding and plugging the next hole. But it misses a category of risk that doesn't go away no matter how good the checklist is.

The piece I read, 'Omnipresent availability risks in cloud software,' flips that assumption. It lays out four risks that are built into running distributed systems at scale: resource saturation, network partitions, security controls that block legitimate users, and uncommon operational changes. None of these are one-off mistakes. They are omnipresent. They exist in the background of every large system, waiting for the right combination of load, latency, and configuration to turn them into an incident. You can't eliminate them by being more careful. You can only design around them.

That reframing changes how I think about reliability work. If a class of failures is unavoidable, then the goal isn't a perfect checklist that prevents every failure. The goal is to accept that some failures will happen and to plan for what happens next. That means designing for graceful degradation and fast recovery. It means asking what the system does when a partition occurs, or when a security control locks out a legitimate user, or when a service saturates its resources. The answer can't be 'that shouldn't happen.'

The rare operational change is the one that worries me most. When a procedure happens frequently, operators build muscle memory. They know the steps, they recognize the failure modes, they can react quickly when something goes wrong. But when an operational change happens only a few times a year, or once a year, nobody has that muscle memory. The runbook is a document, not a practiced routine. That's where incidents find you: in the gap between what the runbook says and what actually happens when you execute it under pressure.

That doesn't mean process and runbooks are worthless. They matter, but they matter differently. Instead of trying to prevent every failure, they should be built to support recovery. A checklist is less useful as a guarantee that nothing will go wrong and more useful as a guide for when things do go wrong. The difference is subtle but important: you're not writing a script for a perfect run, you're writing a fallback for an imperfect one.

So next time I'm looking at a system, I'll watch for the runbook that has only been practiced once, or not at all. That is the most likely place for the next incident. Not because anyone was careless, but because the risk is omnipresent and the recovery path is unpracticed. The more useful posture is not to promise that we will never fail, but to make sure that when we do fail, the system degrades predictably and the people on call can recover quickly.

Originally covered at surfingcomplexity.blog ↗