p99 Is Lying to You

March 11, 2026latencymeasurement

Here is a benchmark that reports a p99 of 12 ms for a service that froze for two full seconds during the run. The benchmark isn't buggy. It measured exactly what it observed. It just never observed the part that mattered.

How the tail disappears

Most load generators are closed-loop: a fixed number of workers, each sending a request, waiting for the response, then sending the next one. It's the obvious design and it's wrong for measuring latency.

Suppose your service stalls for two seconds. During that stall, every worker is blocked waiting on its in-flight request. So the generator sends nothing. When the stall clears, each worker records one slow request and carries on. If you were running a hundred workers at ten thousand requests per second, twenty thousand requests should have been issued during that stall. A hundred were. The other nineteen thousand nine hundred — the ones that would each have recorded somewhere between zero and two seconds of waiting — simply do not exist in your histogram.

Gil Tene named this coordinated omission: the load generator has quietly coordinated with the system under test to only ask questions at moments when the system is ready to answer.¹

Real users do not wait their turn

A user who clicks during your two-second stall waits two seconds. They didn't check whether you were busy first. Load arrives on the world's schedule, not on yours, and a measurement methodology that assumes otherwise is measuring a system nobody uses.

What to do instead

Generate load open-loop: requests go out on a schedule, whether or not previous ones have come back. This requires the generator to be able to have an unbounded number of requests in flight, which is exactly the uncomfortable behaviour you want, because it's what your real clients do.

If you're stuck with a closed-loop tool, correct the histogram. Record each sample along with the interval at which requests were supposed to be issued, and backfill the missing ones — a sample of 2000 ms at an expected interval of 1 ms implies about two thousand synthetic samples decreasing from 2000 ms to 1 ms. HdrHistogram has done this for you for years.

While we're here: you cannot average percentiles

The p99 of ten servers is not the mean of their ten p99 values. It isn't the max either. Percentiles aren't linear and there's no arithmetic that recovers the combined value from the individual ones. If you need a fleet-wide p99, you have to merge the histograms and compute it once. Dashboards that average percentiles across instances are producing a number with no definition.