Each server in a fleet is slow (>1s) just 1% of the time. A user request must wait on 100 of them at once. Roughly how often is the request slow?

Roughly 63% — the rare slowness compounds across all 100

Roughly 1% — each server is rare and independent

Roughly 63% — the rare slowness compounds across all 100

🐢 Why your 99th-percentile latency quietly becomes the median

In Google's classic essay "The Tail at Scale," Jeff Dean and Luiz André Barroso point out a trap that catches almost every large system. Suppose one server responds slower than a second just 1% of the time — a p99 you'd happily ship. Feels rare, right?

Now make a single user request fan out to 100 servers and wait for all of them (a common pattern: search, feeds, anything that queries many shards). The request is only as fast as its slowest piece. The chance that at least one of the 100 is slow is 1 − 0.99¹⁰⁰ ≈ 63%. Your rare 1% event now hits nearly two out of three requests.

The lesson: at scale, tail latency is the system's true latency. That's why big services fight the tail directly — sending duplicate "hedged" requests, setting aggressive timeouts, and routing around slow replicas — instead of chasing a better average that users never actually feel.