This article is related to:
Debugging Lag: Network Connectivity Intro
Debugging Lag: ping and traceroute background info

USING MTR TO COLLECT AND PROVIDE STATISTICS TO THERE

If you have not already read the articles linked to above, please do so first!

Now that you've got a basic understanding of servers, routers, ping, and traceroute, I'd like to suggest a tool that combines ping and traceroute and lets you collect more significant statistics more easily. We will ask that you use this tool when reporting information to us.


The tool is a windows version of a popular unix traceroute called "Matt's Traceroute" or "mtr". You can download winmtr from:

http://prdownloads.sourceforge.net/winmtr/

This is a very small download (134 kb), and it will not introduce any spyware or viruses to your computer. Sourceforge.net is a well respected and safe site. The current version of mtr as I write this is winmtr_bin_0.8.zip. If a newer version has come out by the time you visit this site, it's OK to just download the program (the winmtr_bin_ ... file). You do not need to get the source (winmtr_src_ ... file).

The winmtr .zip file contains WinMTR.exe. Run it, and you will see that you canenter a Host. Enter www.there.com and click Start (or hit Enter).

Let it run for a while! This gives you a nice summary of the number of packets sent/received with the % loss, and also the best round-trip time and the best/average/worst and most recent round-trip times.

Here's an example of an mtr I just ran to www.google.com. Let's look at this one:


|------------------------------------------------------------------------------------------|
|                                      WinMTR statistics                                   |
|                       Host              -   %  | Sent | Recv | Best | Avrg | Wrst | Last |
|------------------------------------------------|------|------|------|------|------|------|
|               firewall-int.thereinc.com -    1 |  334 |  333 |    0 |  130 |  921 |  541 |
|               216.200.151.161.there.com -    2 |  334 |  330 |    0 |  138 | 1101 |  551 |
|                  64.124.60.10.there.com -    1 |  334 |  332 |    0 |  125 |  761 |  491 |
|        64.124.60.62.available.above.net -    2 |  334 |  330 |    0 |  120 |  901 |   60 |
|    262.ge-4-3-0.er10b.sjc2.us.above.net -    2 |  334 |  330 |    0 |  133 |  791 |  100 |
|         so-2-0-0.mpr4.sjc2.us.above.net -    1 |  334 |  331 |    0 |  112 |  651 |   20 |
|          so-2-2-0.cr1.dfw2.us.above.net -    1 |  334 |  332 |   50 |  151 |  811 |  411 |
|          so-0-0-0.cr2.dfw2.us.above.net -    1 |  334 |  333 |   50 |  156 |  942 |  300 |
|          so-5-3-0.cr2.dca2.us.above.net -    4 |  334 |  322 |   90 |  595 | 1672 |  140 |
|         so-5-0-0.mpr2.iad1.us.above.net -    1 |  334 |  333 |   80 |  182 |  832 |  380 |
|         so-3-0-0.mpr2.iad5.us.above.net -    1 |  334 |  331 |   80 |  180 |  901 |  361 |
|     216.200.151.110.available.above.net -    2 |  334 |  330 |   80 |  178 |  892 |  290 |
|                          216.239.46.250 -    1 |  334 |  332 |   80 |  182 |  691 |  290 |
|                           72.14.236.179 -    1 |  334 |  331 |   80 |  178 | 1132 | 1132 |
|                           216.239.48.21 -    1 |  334 |  332 |   80 |  174 | 1021 |  531 |
|                          216.239.49.214 -    1 |  333 |  332 |   80 |  191 | 1192 |  541 |
|                           64.233.161.99 -    0 |  333 |  333 |   80 |  202 | 1052 |  530 |
|________________________________________________|______|______|______|______|______|______|
   WinMTR - 0.8. Copyleft @2000-2002 Vasile Laurentiu Stanimir  ( stanimir@cr.nivis.com )
 
There are few things worth noting here. First, I let it run for more than 300 packets sent. That's a pretty good sample size. You'll always want to run for at least 100 packets. Second, you'll note that there are several hops that show a small amount of packet loss. However, the FINAL host listed, which is the destination, has 0% loss. Remember back to what I was explaining earlier about how ping and traceroute use icmp. While route
rs will drop icmp packets, server hosts rarely do. (They can, it's just much, much less common). You will often see small amounts of packet loss from routers along the way, when you don't see loss from the destination host itself. Another thing to keep in mind is that sometimes the packet loss statistics from hops further along is actually caused by problems from closer to you. Finally, my average round-trip time, or latency, to www.google.com, is around 200ms. That's not bad.

Here's an interesting example from my home to www.there.com:  (Note that this is formatted differently because it was run on unix, but it is from the same program, mtr)

                                           Packets               Pings
Hostname                                %Loss  Rcv  Snt  Last Best  Avg  Worst
 1. er1.sfo1.speakeasy.net                25%   95  127    34    9   14    143
 2. 110.ge-0-0-0.cr1.sfo1.speakeasy.net    0%  127  127     9    7    9     31
 3. ge-5-3-0.mpr3.pao1.us.above.net        0%  127  127     9    8    9     57
 4. so-6-1-0.mpr3.sjc2.us.above.net        0%  127  127    10    9   10     29
 5. so-2-0-0.er10a.sjc2.us.above.net       0%  127  127   194    8   24    194
 6. 64.124.33.93.available.above.net       0%  127  127    10    8    9     10
 7. www.there.com                          0%  127  127     9    8    9     11

You can see that my connection to my ISP, speakeasy, is having a lot of packet loss right now, in terms of icmp packets to the router. However, since I'm not getting any packet loss to anything past that, I can rest assured that my important tcp/ip traffic is likely not affected at all. Also I'm seeing very good latency, as there are only a few router hops between my home and www.there.com, and the connection between speakeasy and above.net is likely a pretty good one.

When to report statistics to us:

If your traceroute output shows packet loss > 5% or high latency (>100ms) in the first hop or two, or in hops that are part of your internet service provider's network, there may be a problem with your connection to your ISP or with your ISPs network. If this is what you are seeing, you will need to contact your ISP directly. You may be able to help them understand and diagnose the problem by providing them with the traceroute stati
stics you have collected. In this case, you do not need to report any statistics to us, as we will not be able to help you improve the situation. Note that I'm considering "high latency" to be only 100ms in hops this close to you. In most cases your first hop should have <50ms latency. This might not be true if you have a slow dialup connection.

If your traceroute output shows packet loss > 5% or high latency > 500ms at the point where the router names change from other networks to above.net, then you should report the statistics to us. This may indicate a problem with a peering relationship between the two internet services. We will report that information to Abovenet, and we will also suggest that you report the information to your ISP. To set your expectations in this so
rt of situation, this kind of problem is often due to the two networks not having enough bandwidth at their peering point. Often this will not be changed unless the problem goes on for a long time and users on both sides submit many complaints. So it will be important to report this on both sides. We will let you know of any response we receive from Abovenet.

If your traceroute output shows packet loss > 5% or high latency > 500ms in the Abovenet network, or close to our servers, please report these statistics to us. We will investigate and talk to Abovenet about the problem. We will let you know any information or status update we are able to learn about this kind of situation.

How to report statistics:

Once your winmtr program has collected statistics for at least 200 sent packets, click "Copy Text to clipboard", then paste the text into a post in this forums thread:

http://forums.prod.there.com/forums/showflat.pl?Cat=&Board=problems&Number=452437&page=0&view=collapsed&sb=5&o=0&fpart=

What we'll do with them:

At least once a week for now, we'll see what's new in this thread. If there are too many posts to respond to individually, we'll still gather all of the info, and make a post letting you know how we're following up, and if we've gotten any information or responses from previous followups.

Thanks,
--jessica
There Operations

This article is related to:
Debugging Lag: Network Connectivity Intro
Debugging Lag: ping and traceroute background info