Why a good server sometimes has to say "no" — fast

It feels fair to accept every request that arrives: queue them all, process them in order, drop nothing. But in the Amazon Builders' Library post "Using load shedding to avoid overload," David Yanacek explains why that instinct quietly destroys a system. When a server takes on more work than it can handle, requests pile up in queues, latency climbs, clients hit timeouts and retry — which adds even more load. The server ends up burning all its CPU on work whose answers arrive too late to matter. This collapse is so common it has a name: congestion collapse.

The key distinction is between throughput (requests you attempt) and goodput (requests you actually complete usefully, before the client gives up). Past a certain offered load, throughput keeps rising while goodput falls off a cliff.

Tap to zoom

Load shedding is the fix: when overwhelmed, the server rejects excess requests quickly and cheaply (e.g. an instant 503) instead of queuing them. Rejecting fast costs almost nothing, so the server preserves capacity to fully serve the requests it does accept. Counter-intuitively, doing less total work lets it complete more useful work — the goodput curve flattens into a plateau instead of crashing. The subtle trap: shedding only helps if saying "no" is far cheaper than saying "yes."