Writing

Google explains why its London backup generators failed

Redundant power feeds and backup generators should keep a data centre running through a power failure. Google’s London outage shows why that expectation needs more than an equipment inventory behind it.

First, one feed was already under repair when the other failed. The relevant test wasn’t normal operation; it was recovery during maintenance. For platform teams, that’s the difference between testing database failover with every replica available and testing it while one is offline.

Second, a voltage-sensing flaw and controller misconfiguration prevented the generators from starting. Battery backups ran out before power could be restored. Backup capacity existed, but the activation path failed.

The real shift is... from counting redundant components to validating the complete recovery sequence under degraded conditions. I’d want that evidence before treating a failover plan as production-ready.

Source: https://assets.theregister.com/2026/11/05/20265/?utm_source=tldrit

Originally covered at assets.theregister.com ↗