Routing/latency changes with "new" network

Forums › 2degrees (including Slingshot, Orcon, Flip, Stuff Fibre, MyRepublic, 2talk and Vocus) › Routing/latency changes with "new" network

1 | 2

openmedia

3324 posts

Uber Geek

Trusted

#3377263 27-May-2025 14:34

yitz:

openmedia:

Looks like Orcon have a fresh set of routing issues via feenix.co.nz. I can't get to a number of sites for the last couple of days including mightyape. Is anyone else with Orcon or 2degrees seeing the same issue?

Did you mean to post in this thread? https://www.geekzone.co.nz/forums.asp?forumid=81&topicid=318294

Would be interesting to know whether it was the filter infrastructure playing havoc with website authentication...

I picked up on this thread as it mentioned the peering with feenix which appears to be part of the packet loss I'm seeing.

Generally known online as OpenMedia, now working for Red Hat APAC as a Technology Evangelist and Portfolio Architect. Still playing with MythTV and digital media on the side.

MadEngineer

4271 posts

Uber Geek

Trusted

#3377336 27-May-2025 19:32

^ packet loss to where?

openmedia:

Looks like Orcon have a fresh set of routing issues via feenix.co.nz. I can't get to a number of sites for the last couple of days including mightyape. Is anyone else with Orcon or 2degrees seeing the same issue?

t14s.local (192.168.0.57) -> www.mightyape.co.nz (151.101.130.133) 2025-05-27T13:59:27+1200
Keys: Help Display mode Restart statistics Order of fields quit
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. _gateway 0.0% 16 0.6 0.6 0.5 0.8 0.1
2. default-rdns.vocus.co.nz 0.0% 16 1.3 1.7 1.3 2.3 0.3
3. default-rdns.vocus.co.nz 0.0% 16 2.6 2.2 1.8 3.1 0.4
4. default-rdns.vocus.co.nz 0.0% 16 13.8 14.2 13.5 17.6 1.0
5. (waiting for reply)
6. 124.150.165.1 0.0% 16 14.9 14.7 14.2 15.6 0.4
7. 124.150.165.2 0.0% 16 15.5 15.2 14.5 16.7 0.5
8. (waiting for reply)
9. 223-26-27-8.feenix.co.nz 92.9% 15 14.9 14.9 14.9 14.9 0.0
10. as54113.akl.ix.nz 0.0% 15 15.6 17.0 14.4 37.6 5.8
11. 151.101.130.133 0.0% 15 14.6 15.6 14.5 20.5 1.5

That 9th hop could just dropping your pings to prioritise web traffic. Mightyape's webserver isn't dropping anything. The pings on a traceroute for those from yourself to that hop, and it's up to each hop how it responds back to you. The pings are not a test of how it would forward your traffic to the next hop

How exactly is that transpiring as a fault to you? Is mightyape slow? Frequently failing to load? Images taking forever to display?

You're not on Atlantis anymore, Duncan Idaho.

arjoll

22 posts

Geek

Subscriber

#3377337 27-May-2025 20:01

openmedia:

Looks like Orcon have a fresh set of routing issues via feenix.co.nz. I can't get to a number of sites for the last couple of days including mightyape. Is anyone else with Orcon or 2degrees seeing the same issue?

Yes, although it wasn't the original topic of the thread - that was me losing my sweet 1ms ping to the office. I also had weirdness with Ring cameras not working when I was using IPoE with 2degrees after being moved from Snap to Slingshot and not being able to authenticate with the old southnet.co.nz PPP username, but any random snap.net.nz one working.

My wife noticed that she couldn't get on to Mighty Ape a week or so ago, and I've run some tests and it's happening on any device on our network. I hadn't done enough tests to post here or contact 2degrees though, but interesting that you're getting the same problem.

This is from home over 2degrees (Slingshot):

Tracing route to mightyape.co.nz [151.101.130.133]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms 10.2.76.254
2 4 ms 4 ms 5 ms v1.cpcch-ric-bng1.tranzpeer.net [101.98.0.113]
3 27 ms 26 ms 27 ms default-rdns.vocus.co.nz [101.98.5.94]
4 37 ms 37 ms 37 ms default-rdns.vocus.co.nz [101.98.5.93]
5 * * * Request timed out.
6 37 ms 37 ms 38 ms 124.150.165.1
7 39 ms 39 ms 38 ms 124.150.165.2
8 * * * Request timed out.
9 * 39 ms 38 ms 223-26-27-8.feenix.co.nz [223.26.27.8]
10 41 ms 37 ms 36 ms as54113.akl.ix.nz [43.243.21.65]
11 39 ms 38 ms 38 ms 151.101.130.133

And this is from the office after changing SD-WAN rules to send out this traffic on 2d (Snap) instead of round-robin with them and Voyager:

Tracing route to mightyape.co.nz [151.101.2.133]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms 10.1.76.254
2 2 ms 1 ms 2 ms 244.7.69.111.static.snap.net.nz [111.69.7.244]
3 12 ms 11 ms 10 ms default-rdns.vocus.co.nz [101.98.5.217]
4 11 ms 12 ms 12 ms default-rdns.vocus.co.nz [101.98.5.216]
5 26 ms 27 ms 26 ms default-rdns.vocus.co.nz [101.98.5.94]
6 38 ms 39 ms 41 ms default-rdns.vocus.co.nz [101.98.5.93]
7 * * * Request timed out.
8 39 ms 38 ms 39 ms 124.150.165.1
9 40 ms 41 ms 40 ms 124.150.165.2
10 * * * Request timed out.
11 38 ms 39 ms 38 ms 223-26-27-8.feenix.co.nz [223.26.27.8]
12 63 ms 64 ms 63 ms 202.77.88.31
13 64 ms 62 ms 63 ms 202.77.88.39
14 65 ms 66 ms 65 ms 151.101.2.133

So it's taking the same route after initially bouncing around some more parts of the Vocus network, so I don't think it's an issue with Vocus' peering.

I'll do more testing before posting any requests here, but I think it is probably OT for this thread! I was meaning too earlier, but we got a very, very active 14 (now 16) week old Maine Coon kitten two weeks ago so locking myself away in the home office checking packet captures and tweaking settings on the Firebox after my wife getting mauled all day would NOT be popular.

yitz

2074 posts

Uber Geek

#3377341 27-May-2025 20:44

At 101.98.5.94/93 it's heading toward the DIA filter. That's why I linked to that thread. I thought upstream of the filter they had a policy of not routing traffic back into the domestic internet exchanges due to the potential for asymmetric routing issues maybe the Feenix guy on here would like to comment.

noroad

949 posts

Ultimate Geek

Trusted

#3377358 28-May-2025 07:59

openmedia:

I picked up on this thread as it mentioned the peering with feenix which appears to be part of the packet loss I'm seeing.

If by packet loss you mean packets discarded in the MTR that is not packet loss. The 7750 (Nokia) has limits set in ICMP to protect its processors but is passing packets just fine.

arjoll

22 posts

Geek

Subscriber

#3377362 28-May-2025 08:24

arjoll:

So it's taking the same route after initially bouncing around some more parts of the Vocus network, so I don't think it's an issue with Vocus' peering.

Now that I re-read my reply this morning, I see that of course that's not actually the case. After the initial hops within 2degrees from the Snap network to the Slingshot one, it is the same to feenix then the route from home, where Mightyape doesn't load, is:

9 * 39 ms 38 ms 223-26-27-8.feenix.co.nz [223.26.27.8]
10 41 ms 37 ms 36 ms as54113.akl.ix.nz [43.243.21.65]
11 39 ms 38 ms 38 ms 151.101.130.133

But from the office, where it is working, it is:

11 38 ms 39 ms 38 ms 223-26-27-8.feenix.co.nz [223.26.27.8]
12 63 ms 64 ms 63 ms 202.77.88.31
13 64 ms 62 ms 63 ms 202.77.88.39
14 65 ms 66 ms 65 ms 151.101.2.133

I don't think this is my end of things!

Home office is via a WatchGuard Firebox T70 with expired security suite (so no web filtering, one day I'll have to talk to Kurt about a new NFR) on 2degrees/Slingshot gigabit residential with PPPoE (IPoE breaks my Ring cameras), while work is via a Firebox M290 with total security on 2degrees/Snap and Voyager, both gigabit business, separate ONTs. For testing I set the SD-WAN rules to 2degrees only.

openmedia

3324 posts

Uber Geek

Trusted

#3377364 28-May-2025 08:29

MadEngineer:

^ packet loss to where?

[How exactly is that transpiring as a fault to you? Is mightyape slow? Frequently failing to load? Images taking forever to display?

Their website simply won't load over my Orcon fibre connection. Works fine over mobile data if I tether my Skinny phone.

I've tried DNS changes, rebooting my router and the main different across the two connections are the massive packet losses for feenix.co.nz.

Firefox error is

The connection has timed out

The server at www.mightyape.co.nz is taking too long to respond.

And under Google Chrome

This site can’t be reached
www.mightyape.co.nz took too long to respond

This also times out on my Android phone if I'm using WiFi but works if I'm on mobile data.

I've reported the issue to Orcon, but yet to see a response.

If I use curl to make a request I do get some limited data back - which I can't paste here, but it mostly warns about a lack of JavaScript and a reference to ct.captcha-delivery.com

Generally known online as OpenMedia, now working for Red Hat APAC as a Technology Evangelist and Portfolio Architect. Still playing with MythTV and digital media on the side.

Equinox IT

Cloud spending continues to surge globally, but most organisations haven’t made the changes necessary to maximise the value and cost-efficiency benefits of their cloud investments. Download the whitepaper From Overspend to Advantage now.

noroad

949 posts

Ultimate Geek

Trusted

#3377372 28-May-2025 09:15

yitz:

At 101.98.5.94/93 it's heading toward the DIA filter. That's why I linked to that thread. I thought upstream of the filter they had a policy of not routing traffic back into the domestic internet exchanges due to the potential for asymmetric routing issues maybe the Feenix guy on here would like to comment.

Asymentric routing is not necessarily an issue in most cases when it comes to IP transit networks. It will cause an issue when you are doing stateful inspection on a firewall (this is an end customer function) but this is obviously not the job of an IP transit wholesale network. Route path selection is done based on a complex set of metrics across NZ, AU and the US where the Feenix network peers, but asymetric routing is not an issue in this environment.

djtOtago

1149 posts

Uber Geek

#3377373 28-May-2025 09:17

I'm with Slingshot and mightyape.co.nz loads fine for me.

tracert mightyape.co.nz

Tracing route to mightyape.co.nz [151.101.194.133]
over a maximum of 30 hops:

1 1 ms 1 ms 1 ms 192.168.68.1
2 3 ms 2 ms 2 ms v1.cpcch-ric-bng1.tranzpeer.net [101.98.0.113]
3 23 ms 23 ms 24 ms default-rdns.vocus.co.nz [101.98.5.94]
4 36 ms 36 ms 35 ms default-rdns.vocus.co.nz [101.98.5.93]
5 * * * Request timed out.
6 36 ms 36 ms 36 ms 124.150.165.1
7 36 ms 37 ms 37 ms 124.150.165.2
8 * * * Request timed out.
9 * * * Request timed out.
10 63 ms 61 ms 61 ms as54113.akl.ix.nz [43.243.21.65]
11 36 ms 47 ms 37 ms 151.101.194.133

openmedia

3324 posts

Uber Geek

Trusted

#3377540 28-May-2025 10:06

Couple of tests

$ tracepath mightyape.co.nz
1?: [LOCALHOST] pmtu 1500
1: _gateway 0.565ms
1: _gateway 0.474ms
2: no reply
3: ext.cpcak4-r1.tranzpeer.net 2.238ms asymm 4
4: no reply
5: no reply
6: ext.cpcak4-r1.tranzpeer.net 1.939ms pmtu 1476
6: 124.150.165.1 22.611ms
7: 124.150.165.2 24.463ms asymm 5
8: no reply
9: no reply
10: no reply
11: no reply
12: no reply
13: no reply
14: no reply
15: no reply
16: no reply
17: no reply
18: no reply
19: no reply
20: no reply
21: no reply
22: no reply
23: no reply
24: no reply
25: no reply
26: no reply
27: no reply
28: no reply
29: no reply
30: no reply

and

traceroute mightyape.co.nz
traceroute to mightyape.co.nz (151.101.130.133), 30 hops max, 60 byte packets
1 _gateway (192.168.0.254) 0.478 ms 0.442 ms 0.511 ms
2 * * *
3 ext.cpcak4-r1.tranzpeer.net (101.98.0.66) 2.141 ms 4.706 ms 4.796 ms
4 default-rdns.vocus.co.nz (101.98.5.93) 14.720 ms 14.983 ms 16.498 ms
5 * * *
6 124.150.165.1 (124.150.165.1) 16.918 ms 16.235 ms 14.583 ms
7 124.150.165.2 (124.150.165.2) 16.883 ms 15.636 ms 15.081 ms
8 * * *
9 223-26-27-8.feenix.co.nz (223.26.27.8) 14.711 ms 14.976 ms 14.189 ms
10 * * *
11 * * *
12 * * *
13 * * *
14 * * *
15 * * *
16 * * *co
17 * * *
18 * * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *
30 *

Generally known online as OpenMedia, now working for Red Hat APAC as a Technology Evangelist and Portfolio Architect. Still playing with MythTV and digital media on the side.

MaxineN

Max
1772 posts

Uber Geek

ID Verified

Trusted

Subscriber

#3377562 28-May-2025 11:05

Hi openmedia.

Can you please try another router? I say this as judging by this thread nobody else is having any issues except you now. If we're using the provided gear (Netcomm still or did you get an Orbi?) then you should rule out your local network by swapping the current router.

You might also be upsetting the WAF just enough to stop your resi IP specifically (as you mention mobile data works fine).

Ramblings from a mysterious lady who's into tech. Warning I may often create zingers.

openmedia

3324 posts

Uber Geek

Trusted

#3377726 28-May-2025 17:54

So Orcon have just responded with the usual boilerplate guidance on making sure I'm not using wifi etc.

At almost the same time the problem disappeared. I have have a considerably shorter route to mightyape and no intermediate packet loss. I've literally changed nothing on my side.

t14s.local (192.168.0.57) -> mightyape.co.nz (151.101.130.133) 2025-05-28T17:51:22+1200
Keys: Help Display mode Restart statistics Order of fields quit
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. _gateway 0.0% 18 0.6 0.8 0.6 2.7 0.5
2. default-rdns.vocus.co.nz 0.0% 18 2.1 1.7 1.1 2.1 0.3
3. default-rdns.vocus.co.nz 0.0% 18 2.6 2.6 2.0 4.2 0.5
4. 167.82.128.172 0.0% 17 25.2 25.2 24.8 25.7 0.3
5. 151.101.130.133 0.0% 17 2.2 2.4 2.0 3.2 0.3

Generally known online as OpenMedia, now working for Red Hat APAC as a Technology Evangelist and Portfolio Architect. Still playing with MythTV and digital media on the side.

arjoll

22 posts

Geek

Subscriber

#3377851 29-May-2025 08:01

MaxineN:

Can you please try another router? I say this as judging by this thread nobody else is having any issues except you now.

I also reported in this thread that I was having issues - I just hadn't done enough diagnosis to be comfortable reporting a fault or opening a new thread. I always like to eliminate anything that I can control before doing that.

It is now working for me as well, with a far shorter route:

Tracing route to mightyape.co.nz [151.101.2.133]
over a maximum of 30 hops:

1 1 ms <1 ms <1 ms 10.2.76.254
2 4 ms 4 ms 3 ms v1.cpcch-ric-bng1.tranzpeer.net [101.98.0.113]
3 * * * Request timed out.
4 25 ms 26 ms 25 ms 151.101.2.133

So there certainly was an issue, and it has now been resolved with no changes to my configuration or theirs.

1 | 2