Wednesday, June 18, 2014

#LatencyTipOfTheDay: You can't average percentiles. Period.

I run into the following situation way too often:

You have some means of measuring and collecting latency, and you want to report on it's percentile behavior over time. You usually capture the data either in some form of high-fidelity histogram, or as raw data (each operation has it's own latency info). You then summarize the data on a per-interval basis and place it in some log. E.g. you may have a log with a summary line per interval, describing something like the 90%/ile, 95%'lie, 99%'lie, 99.9%'lie and Max of all results seen in the last 5 seconds. 

Since raw data is (unfortunately) much more commonly used for this because it's hard to get accurate percentile information from most histograms, you worry about the space needed to keep the full data used to produce this summary over time. So you only keep the summaries but throw away the data.

Then you have this nice log file, with one line per summary interval, and over a run (hour, day, whatever), the log file has all sorts of interval data points for each percentile level, which will often shift rapidly over time. And while this data is useful for "what happened when?" needs, it is terrible for summarizing and getting a feel for the overall behavior of the system during the run. It needs to be further summarized to be comprehensible. 

So how do you produce a summary report on the behavior over longer periods of time (e.g. per hour, or per day, or of the whole run) from such per-interval summary reports? 

Well, here is what I see many people do at this point. Ready for this?

They produce a summary from the summaries. Since they really want to report the 90%'lie, 95%'lie, 99%'lie, 99.9%'lie, and Max values for the whole run, but only have the summary data on a per-interval basis, they average the interval summaries to build the overall summary. 

E.g. The 99%'lie for the run is computed as the average of the 99%'lie report of all intervals in the log. Same for the other percentiles.

Any time you see timeline chart plotting some percentile, and there is an average reported in text to give you a feel for what the overall average behavior of the spiky line you are looking at is, that's what you are most likely looking at...

So what's wrong with that?

At first blush, you may think that the hourly average of the per-5-second-interval 99%ile value has some meaning. you know averages can be inaccurate and misrepresenting at times, but this value could still give you some "feel" for the 99%'lie system, right?


The average of 1,000 99%'lie measurement has as much to do with the 99%'lie behavior of a system as the current temperature on the surface of mars does. Maybe less.

The simplest way to demonstrate the absurdity of this "average of percentiles" calculation is to examine it's obvious effect on two "percentile" values that are hard to hide from: The Max (the 100%'lie value) and the Min (the 0%'lie value). 

If you had the Maximum value recorded for each 5 second interval over a 5000 second run, and want to deduce the Maximum value for the whole run, would an average of the 1,000 max values produce anything useful? No. You'd need to use the Maximum of the Maximums. Which will actually work, but only for the unique edge case of 100%...

But if you had the Minimum value recorded for each 5 second interval over a 5000 second run, and want to deduce the Minimum value for the whole run, you'd have to look for the Minimum of the Minimums. Which also works, but only for the unique edge case of 0%.

The only thing you can deduce about the 99%'lie of a 5000 second run for which you have 1,000 5 second summary 99%'lie values is that it falls somewhere between the Maximum and Minimum of those 1,000 99%'lie values...

Bottom line: You can't average percentiles. It doesn't work. Period.


  1. What about the 99%'ile of those 1000 5 second summary 99%'iles? How close is that to the real overall 99%'ile? I suppose this value is at least more accurate than the average.

  2. This comment has been removed by a blog administrator.

  3. Top tip Gil using the 0% and 100%, thanks for the rant and it allies with my long held belief.

  4. Are you facing problem in Installing AVG Antivirus or configuration for Windows, MacOS, and Android. Call us for an instant remote support. Just dial our AVG Antivirus Technical Support Phone Number toll free 1+866-641-8283 or visit our website at VA Phone Support.

  5. I’m indeed very happy for the great help that Dr. Peterwise rendered to me, I am here to give my testimony about a doctor who helped me in my life. I and my wife was infected with HERPES SIMPLEX VIRUS last 4 years ago, we went to many hospitals for cure but there was no solution, on till one faithful day , we saw a blog whereby Dr. Peterwise cured a woman who also had herpes affection,we (Me and My wife) decided to contact him which we did, he asked us to buy some items, unfailingly we sent him the money he will need in buying the items required , He caste the spell and sent me some herbs. He asked me to go for check-up after 17 days of casting the spell, Luckily for me we were tested herpes negative, now I believe all these Testimonies about him on the internet, he is truly a great man, if you are going through the same problem and you need help you can contact the great Herbalist ( or call/ WhatsApp +2349032731129

  6. replica Cartier watches uk online shop, combining elegant style and cutting-edge technology, a variety of styles of replica Cartier calibre de cartier watches uk online shop, the pointer walks between your exclusive taste style.

  7. Thanks to share this best blog with us. It contain such a best information. photographia

  8. Thanks to share this best information with us. You may visit our website to get more about latest online social media updates. Salim Khan Adopted Kids

  9. Hello! I'm very excited to inform everyone that I'm completely cured from my HSV 1&2 recently. I have used Oregano oil, Coconut oil, Acyclovir, Valacyclovir, Famciclovir, and some other products and it's really help during my outbreaks but I totally got cured! from my HSV with a strong and active herbal medicine ordered from a powerful herbalist and it completely fought the virus from my nervous system and I was tested negative after 12 days of using the herbal medicine. I'm here to let y'all know that herpes virus has a complete cure, I got rid of mine with the help of Dr Osalu and his herbal exploit. Contact him via email: and as well write him on WhatsApp +2348078668950.