What causes increased domestic latency?

Forums › Voyager › What causes increased domestic latency?

Behodar

10411 posts

Uber Geek

Trusted

Lifetime subscriber

#299185 18-Aug-2022 12:42

Before getting onto the actual question, I just want to preface this with "this is curiosity, not a complaint". I'm well aware that I'm never going to notice 5 ms in the real world :)

I'm on fibre in Whakatane. Historically (both with my previous ISP, and when I first joined Voyager) a ping to Auckland such as to 1.1.1.1 would take 7-8 ms. However, some time around Christmas last year, this shot up to 13-14 ms and never returned to the lower value. At the time I just put this down to a cable break somewhere, with the traffic taking a longer route, but it seems that that's not the case as it surely would have been fixed by now.

So, I'm curious about what actually happened here. Is there some clever way to see the 'path' your traffic is taking (e.g. is it going via Taupo or something)? Do people in e.g. Tauranga (Edit: Tauranga's UFF so not a good choice) or Opotiki have similar results? Obviously this isn't terribly important, after all I've waited close to a year before even mentioning it, but it's just something that's been niggling at me for months so I've finally got around to asking the question :)

1 | 2

180 posts

Master Geek

Trusted

Voyager

Lifetime subscriber

#2956042 18-Aug-2022 13:07

Heya @Behodar

In your case, the Whakatane area is serviced by Chorus, who then pulls it back to Hamilton (where the Voyager Handover/ENNI is located) via a Tail Extension Service (TES).
From Hamilton, it goes towards AKL on the Voyager network.
While we have no control over Chorus's path to get it back to Hamilton, we can certainly query it if it's significantly longer than expected.

I'm not able to line up your timeline (Christmas 2021) as to any operations on our network that would increase latency.
Let me double check work history and see if there's anything here that may be related.

Incidentally, 1.1.1.1 may not always be Auckland - it's an anycast IP address that can move around. Cloudflare AKL (who provides the 1.1.1.1 service) did have an issue recently (June-ish) that saw the AKL endpoint drop for a bit. It appears to be resolved now, however.
To get a true understanding of the circuit-only latency, and remove any upstream shenanigans, ping your PPP gateway - in your case: 114.23.3.255

Voyager Internet - Network Monkey

Lenovo

Shop now for Lenovo laptops and other devices (affiliate link).

Behodar

10411 posts

Uber Geek

Trusted

Lifetime subscriber

#2956043 18-Aug-2022 13:12

VygrNetworkMonkey: I'm not able to line up your timeline (Christmas 2021) as to any operations on our network that would increase latency.
Let me double check work history and see if there's anything here that may be related.

Christmas-ish. It was probably around about then, but I'm not sure exactly when it started happening; it could've been a couple of months either side.

VygrNetworkMonkey: Incidentally, 1.1.1.1 may not always be Auckland - it's an anycast IP address that can move around. Cloudflare AKL (who provides the 1.1.1.1 service) did have an issue recently (June-ish) that saw the AKL endpoint drop for a bit. It appears to be resolved now, however.
To get a true understanding of the circuit-only latency, and remove any upstream shenanigans, ping your PPP gateway - in your case: 114.23.3.255

1.1.1.1 was just an example, although probably a bit of a poor one!

PING 114.23.3.255 (114.23.3.255): 56 data bytes
64 bytes from 114.23.3.255: icmp_seq=0 ttl=63 time=14.146 ms
64 bytes from 114.23.3.255: icmp_seq=1 ttl=63 time=14.027 ms
64 bytes from 114.23.3.255: icmp_seq=2 ttl=63 time=13.982 ms
64 bytes from 114.23.3.255: icmp_seq=3 ttl=63 time=14.636 ms

Thanks for the response, but don't spend too much time on this on my account :)

yitz

2052 posts

Uber Geek

#2956053 18-Aug-2022 13:54

May as well be in the South Island, only 2 ms extra 😜 here's connection stats for a round trip between Christchurch UFB and ISP server in AKL.
(not Voyager and over Wi-Fi too..)

PING 203.152.100.249 (203.152.100.249): 56 data bytes
64 bytes from 203.152.100.249: seq=0 ttl=61 time=15.968 ms
64 bytes from 203.152.100.249: seq=1 ttl=61 time=15.979 ms
64 bytes from 203.152.100.249: seq=2 ttl=61 time=16.283 ms
64 bytes from 203.152.100.249: seq=3 ttl=61 time=16.176 ms
64 bytes from 203.152.100.249: seq=4 ttl=61 time=16.034 ms

--- 203.152.100.249 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 15.968/16.088/16.283 ms

Just checked between network elements on the ISP backbone (RIPE probe) and is 13.3 ms AKL-CHC.

I guess there is a penalty going between hops of various network carriers.

Also, I wonder if those Chorus paths are asymmetric? e.g. straight to Auckland one direction, and say via Palmerston North or Welly back.

For trans-Tasman connectivity it is common to see now asymmetric latencies e.g. 23ms Southern Cross, 33 ms Hawaiki = 28 ms average round trip.

VygrNetworkMonkey

180 posts

Master Geek

Trusted

Voyager

Lifetime subscriber

#2956167 18-Aug-2022 16:32

Thanks for the response, but don't spend too much time on this on my account :)

You've sparked my interest!
After a bit of a dig, I've found you're not alone either - a number of services in the Whakatane region are reporting the same latency changes.

Interestingly, yesterday it dropped to 7ms .... for a couple of hours, before bumping back up.

I've dropped a query to my Chorus Account Manager, and will await their response!

Voyager Internet - Network Monkey

Behodar

10411 posts

Uber Geek

Trusted

Lifetime subscriber

#2956347 18-Aug-2022 20:25

VygrNetworkMonkey:

Interestingly, yesterday it dropped to 7ms .... for a couple of hours, before bumping back up.

I've dropped a query to my Chorus Account Manager, and will await their response!

This may interest you:

PING 114.23.3.255 (114.23.3.255): 56 data bytes
64 bytes from 114.23.3.255: icmp_seq=0 ttl=63 time=7.594 ms
64 bytes from 114.23.3.255: icmp_seq=1 ttl=63 time=7.467 ms
64 bytes from 114.23.3.255: icmp_seq=2 ttl=63 time=7.547 ms
64 bytes from 114.23.3.255: icmp_seq=3 ttl=63 time=8.358 ms

I wonder how long it'll last!

Edit: Still fine Friday 16:24. Looks like whatever it was has been fixed :)

Behodar

10411 posts

Uber Geek

Trusted

Lifetime subscriber

#2956657 19-Aug-2022 17:38

Aaand we're back to 14 again.

PING 114.23.3.255 (114.23.3.255): 56 data bytes
64 bytes from 114.23.3.255: icmp_seq=0 ttl=63 time=14.649 ms
64 bytes from 114.23.3.255: icmp_seq=1 ttl=63 time=14.296 ms
64 bytes from 114.23.3.255: icmp_seq=2 ttl=63 time=14.776 ms
64 bytes from 114.23.3.255: icmp_seq=3 ttl=63 time=14.139 ms

yitz

2052 posts

Uber Geek

#2959298 25-Aug-2022 13:17

Before a response from Voyager or Chorus, I thought this article might offer some clues:

https://blog.apnic.net/2020/03/20/providing-new-zealand-with-congestion-free-internet/

My guess is it has to do with availability of fibre routes heading out east and their "congestion free networks" policy.

Here's a previous thread on a similar matter:
https://www.geekzone.co.nz/forums.asp?forumid=39&topicid=272174&page_no=3#2522070

Behodar

10411 posts

Uber Geek

Trusted

Lifetime subscriber

#2959353 25-Aug-2022 15:28

Interestingly I'm getting 7-8 ms to Auckland from work, which I believe is via Spark (and presumably also backhauled via Chorus). I'm pretty sure I'm still getting 13-14 at home and will re-test once I knock off for the day.

Edit: Figured I might as well just work from home for an hour. Yup, 14 ms.

VygrNetworkMonkey

180 posts

Master Geek

Trusted

Voyager

Lifetime subscriber

#2959674 26-Aug-2022 09:50

Behodar:

Interestingly I'm getting 7-8 ms to Auckland from work, which I believe is via Spark (and presumably also backhauled via Chorus). I'm pretty sure I'm still getting 13-14 at home and will re-test once I knock off for the day.

Edit: Figured I might as well just work from home for an hour. Yup, 14 ms.

Voyager utilises the Chorus Tails Extension Service (TES), which aggregates services from surrounding regions to a central location. This saves RSPs from needing to be in every regional handover location (and a lot of cost!).

Through my investigation, I'm fairly certain the TES/Chorus backhaul between your location and the Hamilton aggregation point is causing the additional latency (and fluctuations).
I'm still awaiting information from my Chorus Account Manger on this (they're on leave currently, due back next week - I'll follow up with them when they return)

Larger ISPs, such as Spark, may opt to install a handover into the surrounding regions, which empowers them to utilise their own backhaul network and which path it takes.
Having your own backhaul network negates the Chorus' TES service requirements, especially if they have a more direct path to Auckland etc - which is why your Spark tests may have a lower latency in your case.

Voyager Internet - Network Monkey

VygrNetworkMonkey

180 posts

Master Geek

Trusted

Voyager

Lifetime subscriber

#2964209 7-Sep-2022 13:41

Hokay, I have an official answer from Chorus.

The TL;DR version: They're doing stuff, expect it to be back to normal towards the end of the year.

Chorus' response: ("re-printed" with permission)
"Chorus are part way through a migration of a lot of regional sites to new infrastructure which is causing traffic to use different routes and therefore latency changes will occur. The work is not complete and Chorus are awaiting more equipment deliveries before we can complete extra phases of this work. This means current performance may not be the final result, we should have a better view of that by the end of the year.”

Going by the comments from Chorus a number of regions, (not just the ones I've queried), may have similar fluctuation - so if anyone sees latency changes until the rest of the year, this will likely be the reason.

Tagging @Behodar - to make sure you see this :)

Voyager Internet - Network Monkey

Behodar

10411 posts

Uber Geek

Trusted

Lifetime subscriber

#2964210 7-Sep-2022 13:46

Interesting. Thanks for the response. As previously mentioned I'm not going to notice 5 ms either way, but still good to know :)

Behodar

10411 posts

Uber Geek

Trusted

Lifetime subscriber

#3039050 19-Feb-2023 16:29

For anyone interested: This has returned to normal sometime over the past few days. Given the timing I wonder whether it's related to the East Coast/Hawke's Bay situation (I see that Chorus' outage map has a bit more breathing room now, although there are still a lot of affected people there and I feel for them).

Behodar

10411 posts

Uber Geek

Trusted

Lifetime subscriber

#3045804 4-Mar-2023 17:34

And it's back up in the double digits again. I actually noticed this a couple of days ago but didn't report it at the time.

yitz

2052 posts

Uber Geek

#3046140 6-Mar-2023 00:29

The government funded REANNZ network uses Chorus (and Vodafone) fibre optic paths:

https://weathermap.reannz.co.nz/

The outage of the Rotorua <-> Napier segment from early morning of 14th Feb. to early afternoon of 2nd March coincides with what you have observed.

Behodar

10411 posts

Uber Geek

Trusted

Lifetime subscriber

#3112865 7-Aug-2023 15:32

So, it's almost been a full year since I made this thread, and while it did drop back to normal (7-8 ms) for a while, it's spent more time "high" than "low". In fact, over the past month or so it's been even higher than usual with 16-17 ms to 114.23.3.255 (with similar figures to other NZ addresses).

As we're well past the "end of the [2022] year" mentioned in the Chorus response above, is this expected behaviour?

1 | 2