Saturday, June 21, 2014

#LatencyTipOfTheDay: If you are not measuring and/or plotting Max, what are you hiding (from)?

When you monitor response times or latency in any form, and don't measure Max values for response times or latency, or don't monitor the ones you do measure, you are invariably ignoring a system requirement, and hiding the fact that you are ignoring it.

Monitoring against required behavior


The goal of most monitoring systems is (or should be) to provide status knowledge about a system's behavior compared to it's business requirements, and to help you avoid digging where you don't need to (no need to look into the "cause" of things being "just fine"). Drilling into details is what you do when you know a problem exists. But first you need to know the problem exists, which starts with knowing that you are failing to meet some required behavior.

The reason everyone should be measuring and plotting max values is simple: I don't know of any applications that don't actually have a requirement for Max response time or latency. I do know of many teams that don't know what the requirement is, have never talked about it, think the requirement doesn't exist, and don't test for it. But the requirement is always there.

"But we don't have a max requirement"


Yeah, right.

Whenever someone tells me "we don't have a max time requirement", I answer with "so it's ok for your system to be completely unresponsive for 3 days at a time, right?".

When they say no (they usually use something more profound than a simple "no"), I calmly say "so your requirement is to never have response time be longer than 3 days then..."

They will usually "correct me" at that point, and eventually come up with some number that is reasonable for their business needs. At which point they discover that they are not watching the numbers for that requirement.

So if you have the power to measure this Max latency or response time stuff yourself, or to require it from others, start doing so right away, and start looking at it.

Uncovering insanity


Beyond being a universally useful and critical-to-watch requirement. Max is also a great sanity checker for other values. It's harder to measure Max wrong (although the thing many tools report and display as "max" is a bogus forms of "sampled" max) , and it's really hard to hide from it once you plot or display it.

The first conversations that happen when people start to look at max values after monitoring their stuff for a while without them often start with: "I understand that the pretty lines showing 95%'lie, average, and median all appear great, and hover around 70msec +/- 100msec, and that we've been making them better for months..., but if that's really the case, what the &$#^! is this 7 second max time doing here, and how come it happens several times an hour? And why has nobody said anything about this before? ..."

Apologies


Who knows? You may find that those nice fuzzy feelings the 95%'lie charts have been giving you are well justified. So no harm then.

For those of you who find an uglier truth because I made you look, and don't like what they see, I sincerely pre-apologize for having written this...



17 comments:

  1. Hmm - max can be an useful measure - but it can also be a pita. As performance goes up, latency comes down, if you are using a general purpose operating system, then max starts to measure the rate or soft and hard irqs and not anything useful. I would suggest you are off the mark talking about max/min/percentile and all that stuff. What you need to do is a full cluster analysis. Then work out what is the root cause of each cluster (i.e. why they are different from the others) and then and ONLY then figure out how to improve each cluster separately.

    This is basically, what you have already said: measure what you need to measure. Max is 'what' you need 'why'. Once you have isolated a number of similar events which cluster around max then you can get to why.

    Say hi to Schuetze for me :)

    ReplyDelete
    Replies
    1. I was cured of CANCER with the used of natural herbs. My name is Mella Ken and am from US. I love herbs so much. Most times, injection and drugs are just a waste of time. I was cured 8 months ago, i suffered from CANCER for 3 yrs but with the help of Dr.Sayo herbal medicine, i was cured within few weeks of drinking the herbs he sent to me through courier delivery service. This same doctor also cured my Aunty from FIBROID, as soon as i heard she had FIBROID, i quickly refer her to Dr.Sayo and she was cured too after drinking his herbs.I have referred more than 15 persons to Dr.Sayo and they were all cured from their various illness. Have you taken herbs before?. You have spent so much money on drugs,injections,surgeries etc and yet you have no good result to show for it. Contact Dr. Sayo now, he is a herbalist doctor, i assured you of a cure if you drink his natural herbs. Dr.Sayo have herbs that cures Hiv, Herpes, diabetics, asthma, hepatitis, HBP, STD, cancer, chronic, etc. Contact Dr Sayo through his Email address on: sayoherbalhealer@gmail.com or WHATSAPP/CALL him on+2349012175679 Visit his FB page: https://www.facebook.com/Sayo-Herbal-Healer-100145798345000/    
      also visit website: http://sayoherbalhealinghome.com/ or check blogs: https://naturerootandherbscures.blogspot.com/ Please share the good news to other people once you are cured    

      Delete
  2. This comment has been removed by the author.

    ReplyDelete
  3. I added some beef to the actual blog entry after Alexander Turner's comment and my original reply to it (now removed because it's part of the blog). So if his comment reads out of order, it's my fault, not his...

    ReplyDelete
  4. replica rolex Orologi, che combina uno stile elegante e una tecnologia all'avanguardia, una varietà di stili di replica rolex gmt master ii Orologi, il puntatore cammina tra il tuo stile di gusto esclusivo.

    ReplyDelete
  5. Dr. LUCKY Herbs is a good remedy for herpes Virus , I was a carrier of herpes virus and I saw a testimony on how Dr. LUCKY cure Herpes Virus, I decided to contact him, I contacted him and he guided me. I asked him for solutions and he started the remedies for my health. After he finish he sent me the herbs which i took for 2 weeks before asking me to go for a check up and getting there i could not believe that i was confirm HSV 2 Negative after the test, Today i am so happy because i'm free from herpes disease with the help of Dr. LUCKY. Thank God now everything is fine, I'm cured by Dr. LUCKY herbal medicine, I'm very thankful to God for making it possible. you can reach him on his email, i strongly recommend him to any one out here looking for a cure.Contact Dr LUCKY via email: (drluckyherbalcure@gmail.com)  Or WhatsApp; +2348154637647   

    ReplyDelete
  6. Greatest thanks to Dr Oyagu for his herbal drugs that he prepared for me and when i start using it in just 2weeks i was completely cured and that ended my HERPES SIMPLEX 1&2 DISEASE i am so happy and grateful to Dr Oyagu . after reading about him on a testimony of Jason Cash on a blogger. i knew suddenly Dr Oyagu was the right Doctor to cure my HERPES SIMPLEX 1&2 DISEASE. i discuss with Dr Oyagu and he prepared a herbal medicine for me and when it got sent to me in south korean . i used the herbal medicine and 2weeks and i went to check up again. after 15years of suffering from HERPES SIMPLEX 1&2 at last i am smiling once again. Dr Oyagu also has remedy to others disease like COLD SORES,HIV/AIDS,DIABETES.CANCER,HIGH BLOOD PRESSURE AND MANY MORE. I oblige everyone to contact this powerful herbalist Dr Oyagu and be free from your suffering. contact his WhatsApp line: +2348101755322 or his Email:Oyaguherbalhome@gmail.com

    ReplyDelete
  7. Happiness is all i see now, i was herpes sicker for 6 years but i thank God for directing my footsteps to Dr Emu, he gave me instructions and i followed and in 2 weeks i was completely cured using his medicine, I am now Herpes Negative Contact Dr Emu for more information, He also has other cure to other ailment, HIV AIDS, CANCER, DIABETES, HEART DISEASE, HEPATITIS B/C And also restore broken relationships such as Ex HUSBAND, Ex WIFE, BROKEN MARRIAGES, Any kind of Problem in any relationship can be restore back in 24hours, Just Email: emutemple@gmail.com Whatsapp or call: +2347012841542

    ReplyDelete
  8. First, I want to complement this site administrator for creating this platform I frequently get cold sores(which lasts for 5-6 days on my lips and its painful both physically and mentally) .. As a bodybuilder, I basically get outbreak due to the daily pressure and heat in the body.. Recently I stumbled upon my salvation. I found a herbal remedy to cure hsv completely, in just 4weeks ..

    I got to know about Dr Utu after seeing a lady's testimony online. I reached the doctor and I ordered for herpes herbal medicine. He sent the herbs to me and I started making use of them. In just 2days, the cold sores are cleared. 4weeks after medication, I went to the hospital for test of cold sores and it was completely NEGATIVE.

    By following every prescription, i was able to cure my cold sore in less than 4weeks. 100% natural herbs no after effect.
    Note: Please do let others know if this work for you.
    The doctor's email is drutuherbalcure@gmail.com
    Note : follow all instructions accordingly..

    If you are sure that you have benefited from this African herbs be courageous to inform others.
    Thanks in advance

    ReplyDelete
  9. HERBAL DR EMU WHO PREPARE HERBAL MEDICINE TO CURE ALL KINDS OF DISEASES INCLUDING HERPES DISEASE.   
     I have been battling this Herpes disease for almost 3 years now....I tried all possible means to get cure from my Herpes Disease but all to be in vain until i saw a post in a health forum about a herbal Dr Emu who prepare herbal medicine to cure all kind of diseases including Herpes Disease, at first i doubted if it was real but decided to give it a try...when i contact Dr Emu via his email (emutemple@gmail.com)  write him and reply me explain how the process work so after ordering for the medicine I got it within 3/4 working days through DHL Delivery and I took it according to the way Dr Emu instructed, I was so happy after 2 week I took the medicine there was very big change in my health when I was done with the process I go for test, I found out I am negative...Herpes patients should also get in touch with this herbalist Dr Emu to get rid of these Herpes Virus forever his whatsapp number +2347012841542. Website: https://emutemple.wordpress.com/ 

    ReplyDelete
  10. Am Here To Give My Testimony About A Doctor Who Helped Me In My Life. I Was Infected With HERPES SIMPLEX VIRUS In 2019, I Went To Many Hospitals For Cure But There Was No Solution, So I Was Thinking How Can I Get A Solution Out So That My Body Can Be Okay. One Day I Was In The River Side Thinking Where I Can Go To Get Solution. So A Lady Walked To Me Telling Me Why Am I So Sad And I Open Up All To Her Telling Her My Problem, She Told Me That She Can Help Me Out, She Introduce Me To A Doctor Who Uses Herbal Medication To Cure HERPES SIMPLEX VIRUS And Gave Me His Email, So I Mail Him. He Told Me All The Things I Need To Do And Also Give Me Instructions To Take, Which I Followed Properly. Before I Knew What Is Happening After Two Weeks The HERPES SIMPLEX VIRUS That Was In My Body Got Vanished . So If You Are Also Heart Broken And Also Need A Help, You Can Also contact Him Through Dr. Oyagu reach him oyaguherbalhome@gmail.com visit his website https://oyaguspellcaster.wixsite.com/oyaguherbalhome Or whatsapp him via +2348101755322

    ReplyDelete
  11. To know more about Chief Dr Lucky you can visit his website (https://chiefdrluckyherbaltherapy.wordpress.com/)
     
     A friend that suffered from Herpes and was cure with the help of this great herbal doctor Chief Lucky so i decided to contact him for help in getting rid of my families genital herpes virus 1/2 which i did and all i was told to provide was just some useful information and some materials used in preparation of the natural cure and that i did and now i am the happiest person on earth because i am writing this testimony of mine with joy and happiness in my heart to the whole world to know that natural remedy is real because i am a living testimony of Chief Lucky traditional herbal cure and i want you all to contact Chief Lucky via his email: chiefdrlucky@gmail.com or WhatsApp him +2348132777335 am sure he will help you too.

    ReplyDelete
  12. I want to share to the world how Dr Maggi helped me cure my Herpes with his herbs i never believe it will work because i was introduce to him by my friend who was cure of herpes by Dr Maggi, so i will tell you all who is looking for a cure to their herpes, he took research before he could finally get the solution to it. thank God today am a beneficiary to these cure and i went back to the hospital after 7days of taking the herbs and i was tested negative, and he also cure any kind of  cancer without any surgery and he heal with natural herb. please i urge you to contact him now through his email address: drmaggiherbalhome@gmail.com or call/whatsApp him on +19179796387

    ReplyDelete
  13. I was diagnosed as HEPATITIS B carrier in 2013 with fibrosis of the
    liver already present. I started on antiviral medications which
    reduced the viral load initially. After a couple of years the virus
    became resistant. I started on HEPATITIS B Herbal treatment from
    ULTIMATE LIFE CLINIC (www.ultimatelifeclinic.com) in March, 2020. Their
    treatment totally reversed the virus. I did another blood test after
    the 6 months long treatment and tested negative to the virus. Amazing
    treatment! This treatment is a breakthrough for all HBV carriers.

    ReplyDelete
  14. Very interesting subject, appreciate it for posting.

    my web site :부산오피

    ReplyDelete
  15. I was a HIV-AIDS patient and I got it from cheating on my wife. It was sort of a payback but a week later I was told by a friend that the person who I cheated with had the HIV-AIDS virus and did not tell me. I was so stupid by not using a condom I thought since he was an old school friend he was trustworthy. But I was wrong. I cried and cried. Two days later, I got a phone call from my friend and he told me about a person who is known by another friend, who can help me. I could not let my wife know what I was going through. I finally got his email address: oseremenspelltemple@gmail.com and I emailed this man my story and he replied me immediately saying i should be calm and told me that everything will be OK. I could not come to terms with what I was hearing but then I concluded it did not matter because I was so broken I just needed help. I was going out of my mind literally. I was confused with what he was telling me, but I listened. He told me about some materials i need to buy that he needed to cast the spell and I said OK. I bought the materials to him, I sent down my picture to him and my positive result sheet and he replied me that i am going to be negative under 3days. I message Him every 2hours for 2day and I knew he thought that I was crazy but I did care I needed a shoulder. Behold, the third day he messaged me i should go for a test that i will be negative. My marriage could be broken because of a stupid mistake and my life was on the line. I remembered when I was going to get the results of my re-test I called him up again and told him that I was going to get my results today and his reply was “so" and that everything will be as he explained. I knew then that he was getting tired of me calling him, maybe I was wrong. Well I got my results and the first person I called was him - again. As he said hello I started to cry and cry. I could not believe it. I was given a second chance in life.This man is a great spiritual HIV/AIDS healer, his healing spell on aids healing is very powerful .please brothers and sister, contact Dr Ose oseremenspelltemple@gmail.com
    whatsapp +2348136482342

    ReplyDelete
  16. How can I explain this to the world that there is a man who can cure HERPES, I was diagnosed for the past 5 mouth I have being into HERPES since 5 mouth, so I decide to look for help in the internet then I found a post write about this great man called Dr Kibaki , people say good thing about him that this man have cured, a lot of people in the internet, him has power's to cure HSV1 AND HSV2 I don’t believe that there is a cure for Herpes Simplex Virus because though there is no cure for herpes that what I have in mind HERPES had no cure well , my HERPES is negative through the using of herbal cured of dr Ojoka, I contacted this man for help because of what i see in the internet. if you need his cured just email him now on drkibakisolutiontemple@yahoo.com , thank you DR. i will never stop shearing you testimony Dr Kibaki . him Can as well CURE THE FOLLOWING DISEASE:-1, HIV/AIDS, 2, Diseases of the hear 3, Malignant tumors 4. Cardiovascular diseases 5. Diabetes mellifluous 6.Influenza and pneumonia 7.Alzheimer's disease 8. hsv1 - hsv2. You can also reach Dr. Kibaki on his email: drkibakisolutiontemple@yahoo.com or call : +1 860 397-2711

    ReplyDelete