It's The Muppet Show!


, posted: 12-Jun-2011 21:25

I got a question in my email this morning asking some interesting questions about traceroute.  I'll reproduce the email along with my response here, in the hope that others might also find it useful.

> I found many times in traceroutes, I see 1st hop latency 200ms, next
> 150ms and by final destination it goes to 100 or sometimes 200 (less
> then or equal to 1st hop).
> I don't understand what it means my 200ms latency on 1st hop itself?
> Also, in most of such traceroutes I found that if we see latency on
> destination that's actually correct and logical but I just don't
> understand middle route at all.

My Reply:

There's a couple of reasons you might be seeing this. First of all, it's important to fully understand how traceroute works.

When you run traceroute, you're actually sending out packets with a small time-to-live. The first packet goes out with a TTL of 1 so when the next hop gets it, the TTL is decremented and the router sends you
back a "Time To Live Exceeded". The IP address of the host that sent back the TTL-exceeded is then recorded and shown to you. Another packet is sent out to the same destination, but with the TTL set to 2. This time the next hop gets it, reduces the TTL by 1 and sends it on. Now when the second hop gets it, the TTL is reduced to 0 and a TTL-exceeded is sent back from the 2nd hop.

That's pretty basic and easy to understand, right? That's how traceroute works.

What people don't realise is that the path that the TTL-exceeded packet comes back to you _might be very different_ than the path it took to get there.

Let's took at a pretend example:

Our source address is and we run a trace:

traceroute to (, 30 hops max, 60 byte packets
1 ( 4.977 ms 5.264 ms 5.625 ms
2 ( 600.603 ms 600.238 ms 600.435 ms
3 ( 8.977 ms 8.264 ms 8.625 ms

That 2nd hop looks strange? But why?
Let's pretend we have access to each of the 3 hops there and can trace back to our source of

From hop 1:

traceroute to (, 30 hops max, 60 byte packets
1 ( 4.977 ms 5.264 ms 5.625 ms

Well that was quick, one hop!

From hop 2:

traceroute to (, 30 hops max, 60 byte packets
1 ( 4.977 ms 5.264 ms 5.625 ms
2 ( 6.894 ms 6.234 ms 6.789 ms
3 ( 600.654ms 600.234ms 600.723ms
4 ( 602.845ms 601.932ms 601.945ms
5 ( 600.654ms 600.234ms 600.723ms
6 ( 602.874ms 601.456ms 601.456ms

Woah! What happened?

The return path that has is very a very slow, long path!
The packet still gets there, but it takes ages because it's routing over let's pretend it's a satellite link.

But then the return path from looks like:

traceroute to (, 30 hops max, 60 byte packets
1 ( 4.977 ms 5.264 ms 5.625 ms
2 ( 4.977 ms 5.264 ms 5.625 ms
3 ( 6.678ms 601.331ms 601.824ms

Nice and fast again. It has it's own fast good paths back to that doesn't have.

Now that's a _very_ extreme example (it pretends each different provider has only one hop) and in the real world you probably won't see things that extreme.

However what I'm trying to point out is that traceroute _only_ ever shows you the outwards path between you and your destination. It doesn't show you (and you can't see) the return path that the TTL-Exceeded packets travel back on. Usually it's the same path. But in an asymmetrical routing situation (what I've  shown above) that's not the case.

The OTHER thing that can cause high TTL's can be a router under load. When most ISP routers forward packets these days, they do so in hardware. However if they have to examine the packet and process it, it has to be handed out of the hardware forwarding plane and be given to the CPU to software process. Now the CPU might be very busy, so it might take it a few tens or even hundreds of milliseconds to process it via the CPU. i.e. longer that it'd have taken it to hardware switch it (a few ms at most) That means that some hops might show a high latency spike but then the rest won't, because in most cases the router is just hardware switching the traceroute packet, except for the time when it has to generate the TTL exceeded packet itself.

Hope that helps explain it. Traceroute is a good tool, but it only shows one side of the packet path, which might be only half the story if there's asymmetrical routing going on.

Be sure to follow me on Twitter.

Other related posts:
Install Your Own Mini Google
Understanding Traffic Flow of a NZ ISP.

Permalink to Traceroute | Main Index

Comment by Ash Dando, on 14-Jun-2011 10:36

Good article! One way around this would be a tool which does the following: 1. traceroute to establish the IPs of the outbound path 2. pings to each hop to collect statistics. As far as I know, this is exactly what MTR does - see (linux) or (WinMTR - for Windows). Great little tool, but... However, even MTR is susceptible to skewed results because of another phenomenon - traffic shaping & queuing mechanisms. Some ISPs de-prioritise ICMP traffic. I.e. in a congestion situation their network may be configured to give traceroute (and/or other ICMP traffic such as ping) the lowest priority. I've had multiple ISPs explain this to me (plus I used to be ops manager for a UK telco/ISP :) ..... ICMP traffic includes traceroute and pings, ergo another reason why traceroute results can be skewiff. Then lastly of course some hosts just don't respond to some or all ICMP traffic as it is a common attack vector used by hackers (or *was*, when I left the ISP industry in 2007) ... cheers Ash

Author's note by muppet, on 14-Jun-2011 10:44

Yea, MTR is a slightly enhanced version of traceroute.  It does a trace then keeps sending pings to the hops.  It still doesn't give you any visability of the path the return packets are taking.

If you're having problems with ICMP being blocked you can use traceroute on a unix box which uses UDP, or use my favourite tool which is TCP traceroute.  Using something like port 25 or port 80 you can get past most firewalls using it.


muppet's profile

New Zealand

My name's Tim and I'm a Network Engineer! Well these days I'm actually a Network Architect. I'm one of those weird people that really love their job. I work with a great bunch of people in an interesting industry that's always changing. Sometimes slower than I'd like.

In my spare time I work on the LiCe Script for the EPIC5 IRC Client. Try it out, it's like Irssi but better.