Saturday, June 21, 2014

#LatencyTipOfTheDay:
Q: What's wrong with this picture?
A: Everything!

Question: What's wrong with this picture:


Answer: Everything!

This single chart (source redacted to protect the guilty) is a great depiction of my last four #LatencyTipOfTheDay posts in one great visual:

1. Average (def): a random number that falls somewhere between the maximum and 1/2 the median. Most often used to ignore reality.

Well, that one is self explanatory, and the math behind it is simple. But I keep meeting people who think that looking at average numbers tells them something about the behavior of the system that produced them... Which is especially curious when what they are trying to do is monitor health, readiness, and responsiveness behavior. This chart summarizes averages of the things it plots. Just like everyone else. I hear that on the average these days, the tooth fairy pays $2/tooth.

2. You can't average percentiles. Period.

See those averages posted (at the bottom of the chart) for the 25%, 50%, 75%, 90%, and 95% lines? It's not just that average numbers are low in meaning in their own right. Averaging percentiles is so silly mathematically that it may be a good way to build a random number generator. Read this #LatencyTipOfTheDay post for an explanation of just how meaningless averages of percentiles are.

3. If you are not measuring and/or plotting Max, what are you hiding (from)?

This is a classic "feel good" chart. The chart is pretty. And it looks informative. But it really isn't. Not unless all you care about is the things that worked well, and you don't want to to show anything about a single thing that didn't work well. The main practical function of such a chart is to distract the reader, and make them look at the pretty lines that tell the story of what the good experiences were, so that they won't ask questions about how often bad experiences happened.

The way a chart like this achieves this nefarious purpose is simple: it completely ignores, as in "does not display any information about" and "throws away all indication of", the worst 5% of server operations in any given interval, or in the timespan charted as a whole.

This chart only displays the "best 95%" of results. You can literally have up to 5% of server side page response times take 2 minutes each, and this pretty picture would stay exactly the same.

Whenever charts that show response time or latency fail to display the Maximum measurements along with lower quantiles (like the 95%, or the median, or even the fuzzy average), ask yourself: "what are they hiding?".

4. Measure what you need to monitor. Don't just monitor what you happen to be able to easily measure.

This chart plots the information it has. Unfortunately the information it has is not the important information needed for monitoring server response behavior...

Monitoring is mostly supposed to be about "this is how bad the bad stuff was, and this is how much bad stuff we had". It's been a while since I had met anyone operating servers that only cared about the fastest operation of the day. Or only the best 25% of operation. Or only about the better half of operations. Or only the better 95% of operations, but didn't care at all about the worst 5% of operations. But that's exactly (and only) what the 25%'lie, median, and 95% lines in this chart display. It is literally a chart showing "this is how good the good stuff was". 

Percentiles matter for monitoring server response times, and they matter to several 9s in the vast majority of server applications. The fact that your measurements or data stores only provided you with common case latencies (and yes, 95% is still common case) is no excuse. You may as well be monitoring a white board.

You see, when you actually ask scary questions like "how many of our users will observe the 99.99%'lie of server side HTTP request time each day?" you'll typically get the scary answer that makes you realize you really want to watch that number closely. Much more so than the median or the 95%'ile. Not asking that question is where the problems start.

For example, the statement "roughly 10% of all users will experience our 99.99%'lie once a day or more" is true when each page view involves ~20 HTTP gets on the average, and the average user does ~50 page views or refreshes during a day. Both of these are considered low numbers for what retail sites, or social networking sites, or online application sites experience, for example. And very few users will "fail to experience" the 99.9%'lie in 50 page views under the same assumptions. So if you care about what a huge portion of your user based is actually experiencing regularly, you really care about 99.9% and 99.99%'iles.

But even though I have yet to meet an ops team that does not need to monitor the 99.9% and/or 99.99%'lie of their server response times, it's rare that I meet one that actually does monitor at those levels. And the reason is that they are usually staring at dashboards full of this sort of feel good chart, which distract them enough to not think about the fact that they are not measuring what they need to monitor. Instead, they are monitoring what they happen to be able to measure...

Once you know what you want to be watching, the measurement and the retention of data is where the focus should start, as those pretty picture things usually break early in the process because the things that need monitoring were simply not being measured. E.g. many measurement systems (e.g. Yammer metrics when using its default reservoir configurations) do not even attempt to actually measure percentiles, and model them statistically instead. Of the ones that actually do measure percentiles on the data they receive (e.g. StatsD, if you sent it un-summarized data about each and every server response time), the percentile measurements are typically done on relatively short intervals and summarized (per interval) in storage, which means two things: (A) You can only measure very low numbers of 9s, and cannot produce the sort of percentiles you actually should be monitoring, and (B) the percentiles cannot be usefully aggregated across intervals (see discussion).

(A) happens because intervals are "short", and cannot individually produce any useful data about "high number of 9s". E.g. there is no way to report useful information on the 99.9%'lie or 99.99%'lie in a 5 second interval when the operation rate is 100 ops/sec.

(B) happens because unless percentile data is computed (correctly) across long enough periods of time, it is basically useless for deducing percentiles across more than a single interval. Collecting the data for producing percentiles over longer periods is relatively straightforward (after all, it's being done for each interval with low numbers of 9s), but has some storage volume and speed related challenges with commonly used percentile computation techniques. Challenges that HdrHistogram completely solves now, BTW.

So what can you do about his stuff? A lot. But it starts with sitting down and deciding what you actually need to be monitoring. Without monitoring things that matter, you are just keeping your operations and devops folks distracted, and keeping your developers busy feeding them distracting nonsense.

And stop looking at feel good charts. Start asking for pretty charts of data that actually matters to your operations...

11 comments:

  1. replica rolex Orologi, che combina uno stile elegante e una tecnologia all'avanguardia, una varietà di stili di replica rolex gmt master ii Orologi, il puntatore cammina tra il tuo stile di gusto esclusivo.

    ReplyDelete
    Replies
    1. I was cured of CANCER with the used of natural herbs. My name is Mella Ken and am from US. I love herbs so much. Most times, injection and drugs are just a waste of time. I was cured 8 months ago, i suffered from CANCER for 3 yrs but with the help of Dr.Sayo herbal medicine, i was cured within few weeks of drinking the herbs he sent to me through courier delivery service. This same doctor also cured my Aunty from FIBROID, as soon as i heard she had FIBROID, i quickly refer her to Dr.Sayo and she was cured too after drinking his herbs.I have referred more than 15 persons to Dr.Sayo and they were all cured from their various illness. Have you taken herbs before?. You have spent so much money on drugs,injections,surgeries etc and yet you have no good result to show for it. Contact Dr. Sayo now, he is a herbalist doctor, i assured you of a cure if you drink his natural herbs. Dr.Sayo have herbs that cures Hiv, Herpes, diabetics, asthma, hepatitis, HBP, STD, cancer, chronic, etc. Contact Dr Sayo through his Email address on: sayoherbalhealer@gmail.com or WHATSAPP/CALL him on+2349012175679 Visit his FB page: https://www.facebook.com/Sayo-Herbal-Healer-100145798345000/    
      also visit website: http://sayoherbalhealinghome.com/ or check blogs: https://naturerootandherbscures.blogspot.com/ Please share the good news to other people once you are cured    

      Delete
  2. Dr. LUCKY Herbs is a good remedy for herpes Virus , I was a carrier of herpes virus and I saw a testimony on how Dr. LUCKY cure Herpes Virus, I decided to contact him, I contacted him and he guided me. I asked him for solutions and he started the remedies for my health. After he finish he sent me the herbs which i took for 2 weeks before asking me to go for a check up and getting there i could not believe that i was confirm HSV 2 Negative after the test, Today i am so happy because i'm free from herpes disease with the help of Dr. LUCKY. Thank God now everything is fine, I'm cured by Dr. LUCKY herbal medicine, I'm very thankful to God for making it possible. you can reach him on his email, i strongly recommend him to any one out here looking for a cure.Contact Dr LUCKY via email: (drluckyherbalcure@gmail.com)  Or WhatsApp; +2348154637647   

    ReplyDelete
  3. FINALLY FREE FROM HERPES VIRUSI thought my life had nothing to offer anymore because lifebecame meaningless to me because I had Herpes virus, thesymptoms became very severe and bold and made my familyrun from and abandoned me so they won't get infected. I gaveup everything, my hope, dreams,vision and job because thedoctor told me there's no cure. I consumed so many drugs butthey never cured me but hid the symptoms inside me makingit worse. I was doing some research online someday when Icame across testimonies of some people of how DR Ebhotacured them from Herpes, I never believed at first and thoughtit was a joke but later decided to contact him on the detailsprovided and when I messaged him we talked and he sent mehis herbal medicine and told me to go for a test after twoweeks. Within 7 days of medication the symptomsdisappeared and when I went for a test Lo and behold I wasNEGATIVE by the Doctor Who tested me earlier. Thank you DREbhota because I forever owe you my life and I'll keep ontelling the world about you. If you are going through samesituation worry no more and contact DR Ebhota viadrebhotasolution@gmail. com or WhatsApp him via +2348089535482.he also special on cureing 1. HIV/AIDS2. HERPES 3. CANCER 4.ALS 5. HEPATITIS B 6.DIABETES 7. HUMAN PAPILOMA VIRUS DISEASE(HPV)8. ALZHEIMER 9. LUPUS (Lupus Vulgaris or LupusErythematosu

    ReplyDelete
  4. FINALLY FREE FROM HERPES VIRUSI thought my life had nothing to offer anymore because lifebecame meaningless to me because I had Herpes virus, thesymptoms became very severe and bold and made my familyrun from and abandoned me so they won't get infected. I gaveup everything, my hope, dreams,vision and job because thedoctor told me there's no cure. I consumed so many drugs butthey never cured me but hid the symptoms inside me makingit worse. I was doing some research online someday when Icame across testimonies of some people of how DR Ebhotacured them from Herpes, I never believed at first and thoughtit was a joke but later decided to contact him on the detailsprovided and when I messaged him we talked and he sent mehis herbal medicine and told me to go for a test after twoweeks. Within 7 days of medication the symptomsdisappeared and when I went for a test Lo and behold I wasNEGATIVE by the Doctor Who tested me earlier. Thank you DREbhota because I forever owe you my life and I'll keep ontelling the world about you. If you are going through samesituation worry no more and contact DR Ebhota viadrebhotasolution@gmail. com or WhatsApp him via +2348089535482.he also special on cureing 1. HIV/AIDS2. HERPES 3. CANCER 4.ALS 5. HEPATITIS B 6.DIABETES 7. HUMAN PAPILOMA VIRUS DISEASE(HPV)8. ALZHEIMER 9. LUPUS (Lupus Vulgaris or LupusErythematosu

    ReplyDelete
  5. Nice post. It is really interesting. Thanks for sharing the post!
    Buy AC Online | Smart LED TV
    Laptops for Sale | Best Inverter AC

    ReplyDelete
  6. Greatest thanks to Dr Oyagu for his herbal drugs that he prepared for me and when i start using it in just 2weeks i was completely cured and that ended my HERPES SIMPLEX 1&2 DISEASE i am so happy and grateful to Dr Oyagu . after reading about him on a testimony of Jason Cash on a blogger. i knew suddenly Dr Oyagu was the right Doctor to cure my HERPES SIMPLEX 1&2 DISEASE. i discuss with Dr Oyagu and he prepared a herbal medicine for me and when it got sent to me in south korean . i used the herbal medicine and 2weeks and i went to check up again. after 15years of suffering from HERPES SIMPLEX 1&2 at last i am smiling once again. Dr Oyagu also has remedy to others disease like COLD SORES,HIV/AIDS,DIABETES.CANCER,HIGH BLOOD PRESSURE AND MANY MORE. I oblige everyone to contact this powerful herbalist Dr Oyagu and be free from your suffering. contact his WhatsApp line: +2348101755322 or his Email:Oyaguherbalhome@gmail.com

    ReplyDelete
  7. My health was horrible before I decided to try the Protocol Of taking Dr Ebhota herbal mixture. I felt there was no hope for my health and I was doubtful to try the Protocol thinking it wouldn’t work because I have visited so many hospital but same result. However, I was convinced by my friend to try the herbal medicine because I wanted to get rid of HPV/WART. The herbal mixture that was given to me was really quick and easy to take, and since I have be taking it for less than 3 days I have less outbreak. But within one week i was fully cured from WART/HPV. The herbal medicine really work and I will like to share this great herb doctor contact with you all email him drebhotasolution@gmail.com or whatsapp +2348089535482. Pls try and help yourself out of warts completely today. he also c My health was horrible before I decided to try the Protocol Of taking Dr Ebhota herbal mixture. I felt there was no hope for my health and I was doubtful to try the Protocol thinking it wouldn’t work because I have visited so many hospital but same result. However, I was convinced by my friend to try the herbal medicine because I wanted to get rid of HPV/WART. The herbal mixture that was given to me was really quick and easy to take, and since I have be taking it for less than 3 days I have less outbreak. But within one week i was fully cured from WART/HPV. The herbal medicine really work and I will like to share this great herb doctor contact with you all email him drebhotasolution@gmail.com o r whatsapp +2348089535482. Pls try and help yourself out of warts completely today. he ure DIABETIES ULCAL CANCER etc.He also told me that he has solution for the flowing.1 Cancer cure2 Diabetes cure3 Ringing ear4 Herpes cure5 Warts cure6 HPV cure7 Get your ex back8 Pregnancy herbal medicine9 Prostate enlargement10 Hepatitis B11 Disability12 Kidney problem Etc.

    ReplyDelete
  8. I was cured of CANCER with the used of natural herbs. My name is Mella Ken and am from US. I love herbs so much. Most times, injection and drugs are just a waste of time. I was cured 8 months ago, i suffered from CANCER for 3 yrs but with the help of Dr.Sayo herbal medicine, i was cured within few weeks of drinking the herbs he sent to me through courier delivery service. This same doctor also cured my Aunty from FIBROID, as soon as i heard she had FIBROID, i quickly refer her to Dr.Sayo and she was cured too after drinking his herbs.I have referred more than 15 persons to Dr.Sayo and they were all cured from their various illness. Have you taken herbs before?. You have spent so much money on drugs,injections,surgeries etc and yet you have no good result to show for it. Contact Dr. Sayo now, he is a herbalist doctor, i assured you of a cure if you drink his natural herbs. Dr.Sayo have herbs that cures Hiv, Herpes, diabetics, asthma, hepatitis, HBP, STD, cancer, chronic, etc. Contact Dr Sayo through his Email address on: sayoherbalhealer@gmail.com or WHATSAPP/CALL him on+2349012175679 Visit his FB page: https://www.facebook.com/Sayo-Herbal-Healer-100145798345000/    
    also visit website: http://sayoherbalhealinghome.com/ or check blogs: https://naturerootandherbscures.blogspot.com/ Please share the good news to other people once you are cured    

    ReplyDelete
  9. *Herpes is a serious and recurring disease which can't be cured through drugs or injections by the American doctors but the best way to deal with Herpes is by taking natural herbs medicine for it and is only few American doctors that know about this herbal medicine from Dr odion.. I have read about Dr odion the great herbalist doctor from African who can cure Herpeewith his powerful herbal medicine. for the people suffering from the following diseases Hiv, Herpes, Cancer, Als,Herpatitis, Diabetes, Hpv,Infections ETC should contact him for his herbal medicine because i am a living testimony and i was cured of Herpes. Although, i sent him what he requested and he sent me his medicine which i took for two weeks and today when i went for test i was tested Herpes negative and he can also cure herpes etc you can reach him through his WhatsApp number:+2349019421176 he will help you out or email drodionherbalhome12@gmial.com*

    ReplyDelete
  10. I am really happy that i have been cured from (HERPES SIMPLEX VIRUS) with the herbal medicine of Dr OGBENI , i have been suffering from this disease for the past 3 years and 4 mouth without solution until i came across the email of this doctor who have cure so many people with his herbal medicine, i also choose to give him a chance to help me and my husband, he told me what to do and i kindly did it, and he gave us his herbal medicine and direct me on how to use it, i also follow his instructions for use and he ask us to go for a check up after 1 week and 5days which i did, to my greatest surprise our result came out as negative, we are really happy that there is someone like this doctor who is ready to help anytime any day. To all the readers and viewers that is doubting this testimony stop doubting it and contact this doctor if you really have one and see if he will not actually help you. i am not a stupid woman that i will come out to the public and start saying what someone have not done for me and i know that there are some people out there who are really suffering and hurting their family just because of these diseases so you can Email him on dr.ogbeniherbalhome@gmail.com OR
    WHATSAPP him on +2348103170329

    Thank you for your time and be happy again

    https://www.facebook.com/Dr-Ogbeni-100937351577518/

    ReplyDelete