Why are latencies to overseas networks so much higher than distance?

Forums › New Zealand Broadband › Why are latencies to overseas networks so much higher than distance?

Publius

276 posts

Ultimate Geek

#110899 18-Oct-2012 16:46

I'm wondering if some clueful poeple here can help me.

I've been looking at a few traceroutes from NZ to various overseas destinations and what been bugging me is the times involved with the undersea cable legs.

For example round-trip tracing from Auckland to Sydney (host905.rimuhosting.com)
5. vocus1.ape.nzix.net 3.4
6. ten-0-2-0-400.bdr01.akl01.akl.VOCUS.net.au 43.1

So the real-world one-way distance would be 19.85ms
According to wolfram|alpha "distance auckland sydney" is 10.1ms
According to Southern Cross Networks themselves the auckland sydney cable leg is 11.38ms.

So 19.85 real world, 11.38 for the distance. We have a missing 8.47ms which is huge!

Can anyone explain why? Don't tell me its routers because i'm measuring the between two routers in a path.
Maybe network equipment we can't see? In this trace its within the Vocus network before it heads across the sea and stays within the Vocus network for a good number of hops after so not likely that its bad network design (ie surely all things within their control).

I should add that this is true for Auckland->US etc, and the above trace is not a problem but an example of a generalisation of why Auckland-Sydney (or any other place) seems twice as far away as it should be.

1 | 2

1948 posts

Uber Geek
Inactive user

#703085 18-Oct-2012 16:51

You could be getting routed via the US depending on which ISP you have and the handover points they have.

Plus it's very likely that you could be getting hidden from some core routers / hops depending on the cleverness they have in their core.

How about listing the ISP you are with and a copy specific trace route's you're taking?

Sharesies

Trade NZ and US shares and funds with Sharesies (affiliate link).

michaeln

235 posts

Master Geek

Trusted

#703099 18-Oct-2012 17:39

plambrechtsen: You could be getting routed via the US depending on which ISP you have and the handover points they have.

Plus it's very likely that you could be getting hidden from some core routers / hops depending on the cleverness they have in their core.

How about listing the ISP you are with and a copy specific trace route's you're taking?

Going via the US would add a lot more than another 8ms.

In respect of the original question:

Bear in mind that traceroute can be a terrible indicator of latency. For a start, sending ICMP messages back is often done at low priority, so is not indicative of the experience of a normal customer packet. Also, the return path may be asymmetric---there is no way to tell from traceroute.

Did you run the test multiple times and take an average? Was it just an outlier?

Have you pinged the relevant hosts? Did you get a similar answer?

Are you sure the host in question is actually in Sydney and not in Melbourne?

jeffreyh

50 posts

Geek

#703111 18-Oct-2012 18:04

you will not see every device that a packet travels through
reply packets have to be forward routed through core routers in order to return a reply to the source.
depending on the exit point the return traffic (reply) may not follow the same path as was said previously
each device adds some (small..very) time to the process
control plane traffic which a trace is has low priority and is affected by how busy a device is doing meaningful stuff like forwarding packets
8ms is NOT huge perspective?

insane

3170 posts

Uber Geek

ID Verified

Trusted

#703112 18-Oct-2012 18:13

Publius: I'm wondering if some clueful poeple here can help me.

I've been looking at a few traceroutes from NZ to various overseas destinations and what been bugging me is the times involved with the undersea cable legs.

For example round-trip tracing from Auckland to Sydney (host905.rimuhosting.com)
5. vocus1.ape.nzix.net 3.4
6. ten-0-2-0-400.bdr01.akl01.akl.VOCUS.net.au 43.1

So the real-world one-way distance would be 19.85ms
According to wolfram|alpha "distance auckland sydney" is 10.1ms
According to Southern Cross Networks themselves the auckland sydney cable leg is 11.38ms.

So 19.85 real world, 11.38 for the distance. We have a missing 8.47ms which is huge!

Can anyone explain why? Don't tell me its routers because i'm measuring the between two routers in a path.
Maybe network equipment we can't see? In this trace its within the Vocus network before it heads across the sea and stays within the Vocus network for a good number of hops after so not likely that its bad network design (ie surely all things within their control).

I should add that this is true for Auckland->US etc, and the above trace is not a problem but an example of a generalisation of why Auckland-Sydney (or any other place) seems twice as far away as it should be.

Yes there are sometimes a LOT more devices involved than what you see in a traceroute, and there are several good reasons for that which won't make an awful lot of sense to 99% of people on this forum.

Also as mentioned there is the whole asymmetric routing thing, so your ICMP traffic could take a different path to actual traffic going end-to-end. Vocus uses Cisco gear, so the routers process ICMP by the supervisor which is also tasked with many other things such as route updates etc, whereas actual traffic is processed in hardware by dedicated ASICs.

Some may say I'm a bit biased as I work for Vocus

Publius

276 posts

Ultimate Geek

#703293 19-Oct-2012 00:16

And so to do calculations for the US to prove the same point:

SCC website quotes (segments C+D+E1) 62.44ms one way actual latency between Auckland-SanJose

[Unnecessary parts of mtr output omitted, ms given is avg of many pings]
mtr host112.rimuhosting.com
4. 121.99.12.0 4.9
5. te4-4.ccr01.lax05.atlas.cogentco.com 188.6

different host, mtr host112.rimuhosting.com
7. 203.97.2.37 21.1
8. te7-3.ccr01.sjc05.atlas.cogentco.com 191.3

So we have here 91.85ms and 85.1ms respectively as one-way real-world latencies between auckland-lax/sjc. Somewhere somehow more than 20ms is being used up which is a huge amount of time.

Again, i'm not saying this is a fault or unusual because it's not, i'm just hoping someone can enlighten me as to why in the real world there is far more latency than can be explained by the just the physical speed-of-light distances between two routers on other sides of an ocean.

Fraktul

836 posts

Ultimate Geek

Trusted

#703316 19-Oct-2012 01:40

I'm not too sure if all your figures are based on the submarine cable operators figures or your own calculations.

As previously mentioned by others there are multiple devices on the traffic path between you and your destination, many of which will not show up on a traceroute. Each of these introduces processing delay, each may also introduce delay due to queuing of traffic and prioritisation (ICMP is typically viewed by operators as lower priority).

In addition to this if you are calculating figures based on straight line and speed of light then you must realise that the speed of light figure most often given is within a vacuum, transmission within other mediums is slower. Also cables rarely run in completely straight lines, sources such as wolfram alpha may be calculating simple straight line distance rather then an actual cable path between locations.

kyhwana2

2553 posts

Uber Geek

#703366 19-Oct-2012 08:41

Remember, ping times are round trip!

PenultimateHop

637 posts

Ultimate Geek

Trusted

#703392 19-Oct-2012 09:36

Publius:Can anyone explain why? Don't tell me its routers because i'm measuring the between two routers in a path.
Maybe network equipment we can't see? In this trace its within the Vocus network before it heads across the sea and stays within the Vocus network for a good number of hops after so not likely that its bad network design (ie surely all things within their control).

1. You are not measuring anything of the sort with a single asymmetric traceroute
2. You are not taking into account any photonic or IP switching delay
3. MPLS ICMP/TTL propagation has differing quirks depending on vendor implementation, and since the destination host is in Brisbane, the latency shown for the entire path will look like it is originating from Brisbane since the intermediate P routers will have to tunnel ICMP back to you via the far-end PE.

Azzura

579 posts

Ultimate Geek

ID Verified

#703418 19-Oct-2012 10:39

Perhaps some national security agency is monitoring all internet traffic in and out of the country....ask Dotcom about his ping.

Damn where is my tinfoil hat...

BarTender

3530 posts

Uber Geek

ID Verified

Trusted

Lifetime subscriber

#703423 19-Oct-2012 10:47

I think perhaps a better question to ask is:

- What is your problem?I assume it's your issue with ICMS Ping times taking longer than you expect which may or may not have anything to do with your actual internet performance as detailed above.
Based on that... The next logical question to ask is:

- Are you prepared to either
- A) Pay more than what you already pay for your current internet service
or
- B) Move to another internet provider paying either more or less
or
- C) Engage a wholesale internet provider to get a faster route to the internet.
or
- D) Purchase your own dedicated international link and organise handover in the US or SYD?

To get what you're actually wanting?

How deep are your pockets and you know you can't change the speed of light or the time which core nodes route traffic?

But that really comes back to the first question.. What is the problem you're trying to solve?

and

Publius

276 posts

Ultimate Geek

#703436 19-Oct-2012 11:19

Not trying to solve or fix anything, just trying to get a better understanding of why it seems like (to me) a large fraction of the latency between our island nation and anywhere else is not simply explained only by our distance and the speed of light in fibre.

For all the talk of PIPE networks building a cable to the US that would be a few ms quicker than Southern Cross, it seems to me that there is other lower hanging fruit for fixing our nations latency problem.

Latency is a big problem for us in NZ as its has an effect on how quick we can download from the rest of the world even if there is unlimited undersea bandwidth.

plambrechtsen

1948 posts

Uber Geek
Inactive user

#703448 19-Oct-2012 11:43

Publius: Not trying to solve or fix anything, just trying to get a better understanding of why it seems like (to me) a large fraction of the latency between our island nation and anywhere else is not simply explained only by our distance and the speed of light in fibre.

For all the talk of PIPE networks building a cable to the US that would be a few ms quicker than Southern Cross, it seems to me that there is other lower hanging fruit for fixing our nations latency problem.

Latency is a big problem for us in NZ as its has an effect on how quick we can download from the rest of the world even if there is unlimited undersea bandwidth.

You're right in one respect that there are many other areas apart from the speed of light over to the US impact our download speeds.

Latency is one part of it, as is using your ISPs DNS server and pulling data off a local CDN, as is checking the wiring in your home to ensure your last hop is performing at its best, as is what has already been said about the unreliability of doing a TraceRoute to measure the true latency of your connection since it uses ICMP rather than TCP/UDP.

IMHO the major impacts on your latency will be in order of impact:

- The physical transport you have to your home, UFB or Cable will have less latency than VDSL, which has less latency than ADSL.
- For xDSL the distance from your home to the Exchange/Cabinet and the quality of your internal wiring and if you have interleaving turn on or off depending on your distance all have a significant impact on latency.
- The quality of your ISPs BNG hardware and handover interconnect from your home to the BNG, and BNG capacity / how over-subscribed they are to their wholesale internet provider.
- The skills of your ISPs techs on how they configure & manage the above BNGs and interconnect.
- Availability of ISP provided CDNs either Google or Akamai or other CDN services so your downloads are local rather than pulling them internationally.
- The speed of light to the US.

Make sure the first 5 are ok... And you're doing the best that can be done.

Publius

276 posts

Ultimate Geek

#703451 19-Oct-2012 11:51

I'm not speaking so much about my connection which is UFB so I have nothing to complain about, i get ~4ms to media.nzherald.co.nz, 2.7ms to www.trademe.co.nz, etc.

What i'm wondering about is will things between NZ-AU or NZ-US get any better latency wise, or is the latency we are all getting (its not just my isp or me in particular its all of us) due to some physical law like speed of light.

What i've calculated is that maybe 2/3rds of the latency between NZ-US NZ-AS (and im measuring just the latency between the two routers in a traceroute which are on either sides of the undersea cable) is due to physical limitations, but ~1/3rd isn't and it surprises me that its as high as that.

It surprises me anyway :)

plambrechtsen

1948 posts

Uber Geek
Inactive user

#703468 19-Oct-2012 12:23

So using your example:

traceroute to host905.rimuhosting.com (117.20.11.52), 30 hops max, 60 byte packets
1 x (x) 0.735 ms 0.713 ms 0.695 ms
2 x (x) 9.962 ms 9.929 ms 9.856 ms
3 x (x) 1.200 ms 1.067 ms 1.114 ms
4 x (x) 1.404 ms 1.177 ms 1.310 ms
5 xe2-1-0.akbr4.global-gateway.net.nz (202.50.232.30) 1.518 ms 1.455 ms 1.663 ms = NZ
6 xe6-0-0.sgbr1.global-gateway.net.nz (202.50.232.10) 25.661 ms 25.654 ms 25.502 ms = AU
7 ae0-10.sgbr2.global-gateway.net.nz (202.50.232.246) 25.564 ms 25.668 ms 25.688 ms
8 * vlan556.52gdc76f02.optus.net.au (59.154.21.137) 27.081 ms *
9 te3-3-3927.sg4.optus.net.au (61.88.241.255) 28.198 ms 28.188 ms 28.278 ms
10 119.225.5.238 (119.225.5.238) 27.518 ms 27.229 ms 27.249 ms
11 bri-sot-tpgn-bras2-ge-0-1.tpgi.com.au (203.29.142.50) 41.831 ms 42.079 ms 41.896 ms
12 bri-sot-tpgn-bras2-ge-0-1.tpgi.com.au (203.29.142.50) 41.859 ms 41.796 ms 41.708 ms
13 220-245-236-170.tpgi.com.au (220.245.236.170) 43.165 ms 43.159 ms 43.678 ms
14 tpg-border.v17.xe4.l1.bne.ozservers.net.au (223.252.0.20) 45.223 ms 45.182 ms 45.611 ms
15 internal-mesh.v478.xe6.h1.bne.ozservers.net.au (223.252.0.108) 45.737 ms 45.722 ms 46.350 ms
16 g1-0-10.core0.bne.ozservers.net.au (117.20.0.22) 45.184 ms 44.539 ms 44.820 ms
17 host905.rimuhosting.com (117.20.11.52) 47.946 ms 47.211 ms 46.153 ms

Using a ICMP Traceroute (tcptraceroute isn't installed on my firewall so I couldn't test with that)

The difference from 1.5 ms and 25.6 ms is..

11.38 * 2 (NZ-AU SSC) = 22.76
25.661 - 1.518= 24.143

So the total difference is 1.383 ms.... which is just a bit under the 1.5ms that you have as a hop to akbr4... So where exactly is the problem?

A lot of this will depend on the ISP you're using and the quality of their network and interconnect.

Talkiet

4689 posts

Uber Geek

Trusted

#703483 19-Oct-2012 12:36

Publius: I'm not speaking so much about my connection which is UFB so I have nothing to complain about, i get ~4ms to media.nzherald.co.nz, 2.7ms to www.trademe.co.nz, etc.

What i'm wondering about is will things between NZ-AU or NZ-US get any better latency wise, or is the latency we are all getting (its not just my isp or me in particular its all of us) due to some physical law like speed of light.

What i've calculated is that maybe 2/3rds of the latency between NZ-US NZ-AS (and im measuring just the latency between the two routers in a traceroute which are on either sides of the undersea cable) is due to physical limitations, but ~1/3rd isn't and it surprises me that its as high as that.

It surprises me anyway :)

The main issue is that unless you are sitting in the operational centre of your ISP with a great understanding of MPLS and how the network has been stitched together, you simply can't know the path your traffic is taking - so while you can build a theoretical A-B model using speed of light in a fibre, you don't know the actual distance or traffic path involved.

There are more bits of equipment involved that you can see from a tracert. Often the paths (if you could see them) would appear non-optimal, but that's one of the prices people pay for having a resilient network.

Sorry I can't be more specific. There's a little bit of optimisation that could be done, but it could easily be at the expense of sensible load balancing across alternate paths.

Cheers - N

Please note all comments are the product of my own brain and don't necessarily represent the position or opinions of my employer, previous employers, colleagues, friends or pets.

1 | 2