A Retry Storm, Start to Finish

May 2, 2026incidentreliability

Nothing went down. That's the part worth sitting with. A service got slower by two tenths of a second and we turned that into an outage all by ourselves.

What happened

At 14:02 an index service shipped a change that added a lookup to its request path. Its p50 went from 40 ms to 240 ms. Its p99 went from 180 ms to 600 ms. No errors, no alerts — it was still serving every request correctly, just slower.

Our client had a 250 ms timeout and retried up to three times, waiting a flat 100 ms between attempts. Those numbers had been fine for two years. They were chosen when p99 was 180 ms, and nobody revisited them, because why would you.

The moment p50 crossed 250 ms, roughly half of all calls started timing out and retrying. Each one became up to four requests. The index service's load doubled, then tripled. More load made it slower. Slower made more requests time out. By 14:23 we were sending it about four times our normal request rate and it had stopped being able to serve any of it.

Two things made it worse than it had to be

The delay was a constant. Every client that timed out at the same moment retried at the same moment, 100 ms later. Instead of a smooth increase in load we produced a square wave, and each crest pushed more clients over the timeout, which recruited them into the next crest.

Retries stacked across layers. The mobile client retried. The API gateway retried. Our service retried. Three layers each willing to try three times is up to twenty-seven requests for one user action. Nobody designed that number. It's just what you get when every layer independently decides to be helpful.

What we changed

Backoff is now exponential with full jitter, which is the only variant that actually spreads load rather than merely delaying the collision:¹

sleep = random_between(0, min(cap, base * 2 ** attempt))

We added a retry budget: a token bucket that allows retries to be at most ten percent of successful requests over a sliding window. When a dependency is genuinely sick, the budget empties in seconds and we stop retrying entirely. This is the single change that would have contained the incident on its own.

Retries now happen at exactly one layer. Everything else fails fast and reports upward.

And timeouts became deadlines. The gateway stamps a deadline on the request; every hop computes its own budget from the time remaining instead of starting a fresh 250 ms clock. A hop with 30 ms left doesn't get to spend 250.

The uncomfortable part

Our dashboards were green for the first eleven minutes. Error rate was zero — every request was eventually succeeding, just after a retry. The signal that would have caught it early wasn't error rate or latency. It was the ratio of outbound requests to inbound requests, which is now on the first dashboard anyone opens.