Periodic routing issues to Megaport NSW?

Forums › 2degrees (including Slingshot, Orcon, Flip, Stuff Fibre, MyRepublic, 2talk and Vocus) › Periodic routing issues to Megaport NSW?

3l3m3nt

112 posts

Master Geek

ID Verified

Trusted

Lifetime subscriber

#304284 20-Apr-2023 15:59

I'm on 2degrees, I believe on their newly acquired Vocus network. I work for an Aussie company from home, and I VPN in for remote work. It disconnects me every hour, and I've been trying to figure out why it's only happening to me.

Turns out, they peer with Megaport NSW which 2degrees does too, so pretty good pings overall, but I can't maintain a connection to them for more than an hour. Packet captures show when a TCP session is terminated, the company's border router is sending a TTL exceeded which is telling me there's a potential routing issue.

I thought about logging a ticket through 2degrees support, but could you imagine explaining this to one of the support people?

Hoping someone on here knows someone who can investigate at a higher level?

Happy to send AS numbers and IP's privately.

Dunedin, NZ
Quic Broadband | Rocket 1G/1G Hyperfibre (Yes, you read that right!)
Dunedin Live Webcam (4K) | Quic Smokepings

Referral Links:
Quic (use R282731EPGJMG on checkout for free setup, and to help me pay for my fast internet addiction)
Contact Energy (use FRTDD2R for $100 credit)

3l3m3nt

112 posts

Master Geek

ID Verified

Trusted

Lifetime subscriber

#3066157 20-Apr-2023 17:42

Just an update on this.

It doesn't appear to be hourly. It's random, and happens up to 2 times an hour. It appears that traffic is very temporarily rerouted via Equinix IX Sydney (or other), then it fails back to Mega IX again.

It would appear there is an issue with Vocus's network into Mega IX Sydney like a link flapping or something...?

Dunedin, NZ
Quic Broadband | Rocket 1G/1G Hyperfibre (Yes, you read that right!)
Dunedin Live Webcam (4K) | Quic Smokepings

Referral Links:
Quic (use R282731EPGJMG on checkout for free setup, and to help me pay for my fast internet addiction)
Contact Energy (use FRTDD2R for $100 credit)

myfullflavour

896 posts

Ultimate Geek

Trusted

Full Flavour

#3066185 20-Apr-2023 19:07

Are you on a public IP with 2degrees?

I have heard of issues about TCP connections being dropped when you're behind the CG-NAT.

3l3m3nt

112 posts

Master Geek

ID Verified

Trusted

Lifetime subscriber

#3066187 20-Apr-2023 19:14

Yeah I bought a static IP with them, so it rules out CGNAT fortunately.

Dunedin, NZ
Quic Broadband | Rocket 1G/1G Hyperfibre (Yes, you read that right!)
Dunedin Live Webcam (4K) | Quic Smokepings

Referral Links:
Quic (use R282731EPGJMG on checkout for free setup, and to help me pay for my fast internet addiction)
Contact Energy (use FRTDD2R for $100 credit)

FailedWOF

45 posts

Geek

#3066816 22-Apr-2023 01:54

Vocus AS9790 Megaport and Equinix interfaces seem OK. 2Degrees in Auckland, 5 minute pings.

Megaport IX AKL NZ 10G

Megaport IX 01 SYD AU 20G

Megaport IX 02 SYD AU 20G

Equinix SYD AU 20G

michaelmurfy

meow
13247 posts

Uber Geek

Moderator

ID Verified

Trusted

Lifetime subscriber

#3066830 22-Apr-2023 09:47

@FailedWOF looks like you’re pasting some images from an internal only Smokeping instance nobody else can see resulting in broken images.

Michael Murphy | https://murfy.nz
Referral Links: Quic Broadband (use R122101E7CV7Q for free setup)

Are you happy with what you get from Geekzone? Please consider supporting us by subscribing.
Opinions are my own and not the views of my employer.

freitasm

BDFL - Memuneh
79288 posts

Uber Geek

Administrator

ID Verified

Trusted

Geekzone

Lifetime subscriber

#3066854 22-Apr-2023 10:55

If that's the case, right-click the images on your page and select copy, then just CTRL-V into a reply box to automatically upload to Geekzone.

FailedWOF

45 posts

Geek

#3066870 22-Apr-2023 11:09

freitasm:

If that's the case, right-click the images on your page and select copy, then just CTRL-V into a reply box to automatically upload to Geekzone.

Odd, as that's exactly what I did. Two options, copy image and copy image address in Chrome. I used copy image. Trying again with snip (and I now see the little upload box flash up, which I don't see when using copy image). The option to edit my previous post seems to have gone, but it may as well be deleted.

@michaelmurfy thanks for the heads up as well.

Vocus AS9790 Megaport and Equinix interfaces seem OK. 2Degrees in Auckland, 5 minute pings.

Megaport IX AKL NZ 10G

Megaport IX 01 SYD AU 20G

Megaport IX 02 SYD AU 20G

Equinix SYD AU 20G

Sharesies

Trade NZ and US shares and funds with Sharesies (affiliate link).

3l3m3nt

112 posts

Master Geek

ID Verified

Trusted

Lifetime subscriber

#3067871 25-Apr-2023 12:38

Yeah not sure what's going on. Work VPN works fine if I proxy it through AWS, rerouting it via a different path. Since you're using smokeping, I thought I'd do the same, but I pumped the granularity up a bit, and damn, there's some weirdness...

After pulling all the peers off peeringdb, these are the results I see..

This is the Dunedin BNG (101.98.0.98). Really stable, no issues there just to rule out my connection.

CHC-IX (103.94.93.3)

WLG-IX (103.94.95.5)

AKL-IX 1 (43.243.21.3)

AKL-IX 2 (43.243.21.79)

Megaport IX AKL (43.243.22.29)

Equinix IX (45.127.172.93)

IX-NSW 1 (218.100.52.94)

IX-NSW 2 (218.100.53.4)

Megaport IX SYD 1 (103.26.68.73)

Megaport IX SYD 2 (103.26.69.3)

Anyway.. there's a strange pattern in there, sorta like in regular intervals the whole network jumps in latency then drops down again. Anyway, without visibility it could be anyone's guess...

Dunedin, NZ
Quic Broadband | Rocket 1G/1G Hyperfibre (Yes, you read that right!)
Dunedin Live Webcam (4K) | Quic Smokepings

Referral Links:
Quic (use R282731EPGJMG on checkout for free setup, and to help me pay for my fast internet addiction)
Contact Energy (use FRTDD2R for $100 credit)

yitz

2079 posts

Uber Geek

#3067895 25-Apr-2023 15:20

Do they coincide with your disconnections to your company server though? Not the best idea to ping router interface IPs are their control plane CPU may be busy and generating ICMP responses may be down the list of priorities. For something on net you could ping their distributed YouTube caches like r1.callplus-akl4.c.youtube.com looks to also be a callplus-wlg2 and callplus-chc1 cluster.

If "company's border router is sending a TTL exceeded" coincides I'd be inclined to think it is something on the company network's end. Is that ICMP generated by a firewall on the company's end or something else? Any other users of the company server experiencing problems in NZ? If you are concerned about route churn Vocus NZ's looking glass listed on their peeringdb page may give you some visibility.

3l3m3nt

112 posts

Master Geek

ID Verified

Trusted

Lifetime subscriber

#3067898 25-Apr-2023 16:14

Yeah you're spot on re the control plane on routers responding to pings as a lower priority. I did think the patterns were interesting though.

What I have seen is maybe a rise of 5 or so ms against the company's routers, then about half an hour later the route churns, which results in it dropping back down 5ms RTT, which causes the disconnection. I'm quite certain it's a route churn causing it.

Nobody else at the company experiences what I do. I'm just lucky! I'll talk with the network guys.

Thanks for the post :)

Dunedin, NZ
Quic Broadband | Rocket 1G/1G Hyperfibre (Yes, you read that right!)
Dunedin Live Webcam (4K) | Quic Smokepings

Referral Links:
Quic (use R282731EPGJMG on checkout for free setup, and to help me pay for my fast internet addiction)
Contact Energy (use FRTDD2R for $100 credit)

3l3m3nt

112 posts

Master Geek

ID Verified

Trusted

Lifetime subscriber

#3067912 25-Apr-2023 17:55

I've added probes in for those three youtube caches..

http://smokeping.dalleyfamily.net/

Dunedin, NZ
Quic Broadband | Rocket 1G/1G Hyperfibre (Yes, you read that right!)
Dunedin Live Webcam (4K) | Quic Smokepings

Referral Links:
Quic (use R282731EPGJMG on checkout for free setup, and to help me pay for my fast internet addiction)
Contact Energy (use FRTDD2R for $100 credit)