Writing

Emergency patches: the four ways to ship critical fixes faster

Emergency patches are how software vendors ship critical fixes faster than their normal release cycle. Normal cycles are built for predictability: changes are bundled, tested, and rolled out in stages. An emergency patch is a deliberate exception to that process—a controlled override that puts a fix in front of users as quickly as possible. Google's Project Zero recently broke down the four ways they do it in a piece called 'How to fix a bug in a fix.' I'm not going to rehash the entire report here; I want to focus on the part that matters for anyone running a production system.

In plain English: when a vulnerability is being actively exploited, you don't wait for the next scheduled update. You need a controlled override that gets the fix out now. The main tension is speed versus stability. Every one of the four mechanisms trades one for the other, and the tradeoff is rarely free. The goal is not to replace the normal process; it's to create a narrow, temporary path that can be used under strict conditions.

Project Zero identifies four distinct mechanisms, and they sit at different layers of the stack: two operate inside your own application or infrastructure, one at the network boundary, and one in the release channel itself.

• Feature flags let you turn off a vulnerable code path without redeploying the whole app.
• Filtering blocks malicious traffic at the network edge before it reaches your code.
• Alternate channels let vendors push a patch through a side door, bypassing slow app store reviews.
• Hotpatching modifies running code in memory, avoiding a restart entirely.

The practical impact is clear. A feature flag can neutralize a zero-day in hours instead of days, but only if that flag already exists in the code. Think of it as turning off a water valve versus replacing all the pipes. The valve is instant; replacing pipes is thorough but slow. A feature flag is like a valve that was installed when the plumbing was first put in. If it's there, you can shut off the flow in minutes. If it's not there, you have to drain the system and cut pipes. Most teams want both: fast mitigation first, stable fix second.

A feature flag can neutralize a zero-day in hours instead of days, but only if that flag already exists in the code.

The feature flag example is the one that sticks with me because it exposes a hidden assumption: the fastest fix is only available if you built the capability before you needed it. You can't conjure a feature flag retroactively when the vulnerable path is already under attack. The same goes for hotpatching—you need the instrumentation and runtime support in place ahead of time. Filtering and alternate channels are less dependent on your own code, but they come with their own operational costs.

None of this means you need all four mechanisms today. The point is to know which ones fit your stack and to have at least one fast path ready before you need it. For anyone running high-volume transactional platforms, this is a solid reminder that rapid response isn't just about finding bugs—it's about having pre-built, low-friction paths to ship a fix when every minute counts. If you have to design the override while the incident is already underway, you've already lost time you can't get back.

Originally covered at projectzero.google ↗