BeCuriouzLearn something new every day
Why 'the spike is over' doesn't mean your system recovers
💡 A system can get stuck in a self-sustaining failure state that persists after its trigger is gone — escaping it often requires dropping load below normal, not just waiting.
You'd expect an overloaded service to heal itself the moment the traffic spike passes. Often it doesn't. Engineer Marc Brooker popularized the idea of the metastable failure — formalized in a 2021 HotOS paper — to explain outages where a system stays broken long after whatever triggered it is gone.
The key idea: a system can have two stable states. One is healthy. The other is a congested state that sustains itself through a feedback loop. A brief trigger — a load surge, a short dependency blip — knocks the system into the bad state. Then clients time out and retry, retries pile more work onto an already-saturated system, which causes more timeouts, which causes more retries. The original trigger is long gone, but the loop keeps the system pinned in failure.
The unsettling consequence: removing the trigger isn't enough to recover. To escape, you usually have to push load below the normal level — shed traffic, disable retries, or drop caches — before the system can climb back to health. Worse, well-meaning safety features can deepen the trap: aggressive retries, and even autoscaling that hammers a cold dependency, are common amplifiers.