Wednesday, June 18, 2014

#LatencyTipOfTheDay: You can't average percentiles. Period.

I run into the following situation way too often:

You have some means of measuring and collecting latency, and you want to report on it's percentile behavior over time. You usually capture the data either in some form of high-fidelity histogram, or as raw data (each operation has it's own latency info). You then summarize the data on a per-interval basis and place it in some log. E.g. you may have a log with a summary line per interval, describing something like the 90%/ile, 95%'lie, 99%'lie, 99.9%'lie and Max of all results seen in the last 5 seconds. 

Since raw data is (unfortunately) much more commonly used for this because it's hard to get accurate percentile information from most histograms, you worry about the space needed to keep the full data used to produce this summary over time. So you only keep the summaries but throw away the data.

Then you have this nice log file, with one line per summary interval, and over a run (hour, day, whatever), the log file has all sorts of interval data points for each percentile level, which will often shift rapidly over time. And while this data is useful for "what happened when?" needs, it is terrible for summarizing and getting a feel for the overall behavior of the system during the run. It needs to be further summarized to be comprehensible. 

So how do you produce a summary report on the behavior over longer periods of time (e.g. per hour, or per day, or of the whole run) from such per-interval summary reports? 

Well, here is what I see many people do at this point. Ready for this?

They produce a summary from the summaries. Since they really want to report the 90%'lie, 95%'lie, 99%'lie, 99.9%'lie, and Max values for the whole run, but only have the summary data on a per-interval basis, they average the interval summaries to build the overall summary. 

E.g. The 99%'lie for the run is computed as the average of the 99%'lie report of all intervals in the log. Same for the other percentiles.

Any time you see timeline chart plotting some percentile, and there is an average reported in text to give you a feel for what the overall average behavior of the spiky line you are looking at is, that's what you are most likely looking at...

So what's wrong with that?

At first blush, you may think that the hourly average of the per-5-second-interval 99%ile value has some meaning. you know averages can be inaccurate and misrepresenting at times, but this value could still give you some "feel" for the 99%'lie system, right?

Wrong.

The average of 1,000 99%'lie measurement has as much to do with the 99%'lie behavior of a system as the current temperature on the surface of mars does. Maybe less.

The simplest way to demonstrate the absurdity of this "average of percentiles" calculation is to examine it's obvious effect on two "percentile" values that are hard to hide from: The Max (the 100%'lie value) and the Min (the 0%'lie value). 

If you had the Maximum value recorded for each 5 second interval over a 5000 second run, and want to deduce the Maximum value for the whole run, would an average of the 1,000 max values produce anything useful? No. You'd need to use the Maximum of the Maximums. Which will actually work, but only for the unique edge case of 100%...

But if you had the Minimum value recorded for each 5 second interval over a 5000 second run, and want to deduce the Minimum value for the whole run, you'd have to look for the Minimum of the Minimums. Which also works, but only for the unique edge case of 0%.

The only thing you can deduce about the 99%'lie of a 5000 second run for which you have 1,000 5 second summary 99%'lie values is that it falls somewhere between the Maximum and Minimum of those 1,000 99%'lie values...

Bottom line: You can't average percentiles. It doesn't work. Period.

14 comments:

  1. What about the 99%'ile of those 1000 5 second summary 99%'iles? How close is that to the real overall 99%'ile? I suppose this value is at least more accurate than the average.

    ReplyDelete
  2. This comment has been removed by a blog administrator.

    ReplyDelete
  3. Top tip Gil using the 0% and 100%, thanks for the rant and it allies with my long held belief.

    ReplyDelete
  4. Are you facing problem in Installing AVG Antivirus or configuration for Windows, MacOS, and Android. Call us for an instant remote support. Just dial our AVG Antivirus Technical Support Phone Number toll free 1+866-641-8283 or visit our website at VA Phone Support.

    ReplyDelete
  5. I’m indeed very happy for the great help that Dr. Peterwise rendered to me, I am here to give my testimony about a doctor who helped me in my life. I and my wife was infected with HERPES SIMPLEX VIRUS last 4 years ago, we went to many hospitals for cure but there was no solution, on till one faithful day , we saw a blog whereby Dr. Peterwise cured a woman who also had herpes affection,we (Me and My wife) decided to contact him which we did, he asked us to buy some items, unfailingly we sent him the money he will need in buying the items required , He caste the spell and sent me some herbs. He asked me to go for check-up after 17 days of casting the spell, Luckily for me we were tested herpes negative, now I believe all these Testimonies about him on the internet, he is truly a great man, if you are going through the same problem and you need help you can contact the great Herbalist (peterwiseherbalhome@gmail.com) or call/ WhatsApp +2349032731129

    ReplyDelete
  6. replica Cartier watches uk online shop, combining elegant style and cutting-edge technology, a variety of styles of replica Cartier calibre de cartier watches uk online shop, the pointer walks between your exclusive taste style.

    ReplyDelete
  7. Thanks to share this best blog with us. It contain such a best information. photographia

    ReplyDelete
  8. Thanks to share this best information with us. You may visit our website to get more about latest online social media updates. Salim Khan Adopted Kids

    ReplyDelete
  9. Hello! I'm very excited to inform everyone that I'm completely cured from my HSV 1&2 recently. I have used Oregano oil, Coconut oil, Acyclovir, Valacyclovir, Famciclovir, and some other products and it's really help during my outbreaks but I totally got cured! from my HSV with a strong and active herbal medicine ordered from a powerful herbalist and it completely fought the virus from my nervous system and I was tested negative after 12 days of using the herbal medicine. I'm here to let y'all know that herpes virus has a complete cure, I got rid of mine with the help of Dr Osalu and his herbal exploit. Contact him via email: drosaluherbalhome@gmail.com and as well write him on WhatsApp +2348078668950.    

    ReplyDelete
  10. Thank you so much for the great and very beneficial stuff that you have shared with us. replica watches india

    ReplyDelete
  11. I am very happy to inform the general public that I am finally cured from Herpes simplex virus recently with the use of herbal medicine, the powerful herbal medicine cured me completely and i was tested negative after usage, I am using this means to inform other's who have the virus that there is a cure for Herpes simplex 1&2 . It is absolutely true. I was cured by Dr Alli, only him I can recognize who cure Herpes perfectly well. I will advice you to contact this great doctor who God has send to put an end to the sorrowful and deadly disease. Contact him via Email  Allispellhelp1@gmail.com or add him on WhatsApp +2348100772528

    ReplyDelete
  12. Highly energetic blog, like it. Read replica watches india buying tips from Datsbrand website.

    ReplyDelete
  13. HELLO EVERYONE.. FEW MUNINETS TO REDY THIS INFOR ON HERPES CURE 2018..
    2017 MY MOTHER WAS DIAGNOSED OF HERPES/ KNOWN AS GENITAL WARTS ,I SPENT A LOT OF MONEY ON HER MEDICATION TILL A POINT I EVEN LOST HOPE,BECAUSE MY MOTHER WAS GRADUALLY DYING AND LOST HER MEMORY TOO, I WAS SO DESPERATE TO GET MY MOTHER BACK TO NORMAL, ONE DAY MY UNCLE WHO LIVES IN LONDON UNITED KINGDOM TOLD ME ABOUT DR OLIHA ,WHO HELPED HIM GET RID OF HERPES /GENITAL WART WITH HERBAL MEDICINE AND HIS HERBAL SOAP ,I WAS SO SHOCKED WHEN HE TOLD ME ABOUT THIS ,ALTHOUGH I NEVER BELIEVE IN HERB BUT, I KEEP TO BELIEVE BECAUSE MY UNCLE CAN'T TELL ME LIES WHEN IT COMES TO HEALTH CONDITION I CONTACTED DR OLIHA VIA HIS EMAIL; OLIHA.MIRACLEMEDICINE@GMAIL.COM , YOU CAN TALK TO HIM VIA CALL OR WHATSAPP MESSENGER ON +2349038382931 , HE REPLIED AND ASK ME TO SEND MY HOME ADDRESS AND MY MOTHER'S DETAIL AND THEN I PURCHASED THE HERBAL MEDICINE,SENT ME THE HERBAL MEDICINE THROUGH COURIER SERVICE, WHEN I RECEIVED THIS HERBAL MEDICINE USED IT FOR 2 WEEKS, AND 4 DAYS OF USAGE THE WARTS FELL OFF, MY MOTHER I NOW TOTALLY CURED AND MY MOTHER IS LIVING FREE AND HAPPY AGAIN. YOU CAN TALK TO DR VIA HIS MOBILE NUMBER OR WHATS APP HIM ON +2349038382931.ALL THANKS TO DOCTOR DR OLIHA  

    ReplyDelete
  14. i want to inform the public how i was cured of HERPES Simplex Virus by a Doctor called Dr.EBHOTA. i visited different hospital but they gave me list of drugs like Famvir, Zovirax, and Valtrex which is very expensive to treat the symptoms and never cured me. I was browsing through the Internet searching for remedy on HERPES and i saw comment of people talking about how Doctor EBHOTA cured them. I Was scared because i never believed in the Internet but i was convince to give him a try because i was having no hope of been cured of HERPES so i decided to contact him on his email, i searched his email on net and i saw a lot of people testifying about his goodness. when i contacted him he gave me hope and send a Herbal medicine to me that i took and it seriously worked for me, am a free person now without problem, my HERPES result came out negative. You can contact him on
    his Email: drebhotasolution@gmail.com or whatsapp him on +2348089535482 and get all your problem solved.
    1Sickness of any kind
    2 Help to CURE Hepatitis B, HIV/AIDs CANCER and BIPOLAR
    3 Divorce, Breakup problem and To Re-unit
    4 Pregnancy problem
    5 Financial problem and Job promotion
    6 To get a good and rich life partner
    7 To Lose your wath
    8 to enlargement of pennis
    9. CANCER
    10. ALS (Lou Gehrig's disease)
    11 Hepatitis B
    12 chronic pancreatic
    13 emphysema
    14 COPD (chronic obstructive pulmonary disease)   

    ReplyDelete