NZ -> US Traffic issue

Forums › 2degrees (including Slingshot, Orcon, Flip, Stuff Fibre, MyRepublic, 2talk and Vocus) › NZ -> US Traffic issue

NickMack

962 posts

Ultimate Geek

Trusted

Lifetime subscriber

#280295 7-Dec-2020 09:39

Hi All,

Traffic destined to the US is taking a long path, I suspect due to a fault or planned work, kicked off around 1am this morning, a bit of packet loss too now that we are in business hours. @Morganbrowne - Are you able to provide any details?

Nick.

https://nick.mackechnie.co.nz | NZ ISP latency monitoring - https://smokeping.thenet.gen.nz

1 | 2

423 posts

Ultimate Geek

Trusted

2degrees

#2617470 7-Dec-2020 12:57

This was an issue with an undersea cable amplifier in the US, has now been resolved so should see traffic back to normal.

Any posts are personal comments and not that of my employer

NickMack

962 posts

Ultimate Geek

Trusted

Lifetime subscriber

#2617496 7-Dec-2020 13:43

Thanks Rob, confirming all good, looks like FB maybe playing with stuff to, may be or maybe not related as it looks like traffic isn't hitting your cache.

Nick

https://nick.mackechnie.co.nz | NZ ISP latency monitoring - https://smokeping.thenet.gen.nz

Lorenceo

904 posts

Ultimate Geek

Trusted

#2617513 7-Dec-2020 14:19

On a similar note, some traffic to Aussie looks to be taking a bit of a scenic route compared to normal:

Tracing route to mirror.internode.on.net [150.101.135.3]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms ER4.lan [192.168.2.1]
2 2 ms 5 ms 3 ms 43.7.69.111.static.snap.net.nz [111.69.7.43]
3 * * * Request timed out.
4 103 ms 103 ms 103 ms 23655.syd.equinix.com [45.127.172.156]
5 102 ms 103 ms 102 ms 4739.syd.equinix.com [45.127.172.20]
6 123 ms 121 ms 121 ms be6.cr2.syd7.on.ii.net [150.101.41.125]
7 124 ms 125 ms 126 ms be17.cr2.adl4.on.ii.net [150.101.33.33]
8 122 ms 122 ms 123 ms 150.101.40.146
9 123 ms 122 ms 123 ms mirror.internode.on.net [150.101.135.3]

Trace complete.

pwner

423 posts

Ultimate Geek

Trusted

2degrees

#2617599 7-Dec-2020 15:10

weird not sure why your taking that path

from Auckland

Tracing route to mirror.internode.on.net [150.101.135.3]
over a maximum of 30 hops:

1 1 ms <1 ms 1 ms 192.168.1.1
2 6 ms 5 ms 6 ms 41.7.69.111.static.snap.net.nz [111.69.7.41]
3 7 ms 5 ms 5 ms 198.18.10.197
4 6 ms 6 ms 6 ms 163.10.69.111.static.snap.net.nz [111.69.10.163]
5 * * * Request timed out.
6 27 ms 29 ms 30 ms 218.100.53.62
7 29 ms 29 ms 30 ms as4739.nsw.ix.asn.au [218.100.52.5]
8 49 ms 48 ms 50 ms be6.cr2.syd7.on.ii.net [150.101.41.125]
9 50 ms 51 ms 49 ms be17.cr2.adl4.on.ii.net [150.101.33.33]
10 50 ms 50 ms 50 ms 150.101.40.146
11 47 ms 51 ms 51 ms mirror.internode.on.net [150.101.135.3]

Trace complete.

Any posts are personal comments and not that of my employer

NickMack

962 posts

Ultimate Geek

Trusted

Lifetime subscriber

#2617686 7-Dec-2020 16:54

pwner:

weird not sure why your taking that path

from Auckland

Tracing route to mirror.internode.on.net [150.101.135.3]
over a maximum of 30 hops:

1 1 ms <1 ms 1 ms 192.168.1.1
2 6 ms 5 ms 6 ms 41.7.69.111.static.snap.net.nz [111.69.7.41]
3 7 ms 5 ms 5 ms 198.18.10.197
4 6 ms 6 ms 6 ms 163.10.69.111.static.snap.net.nz [111.69.10.163]
5 * * * Request timed out.
6 27 ms 29 ms 30 ms 218.100.53.62
7 29 ms 29 ms 30 ms as4739.nsw.ix.asn.au [218.100.52.5]
8 49 ms 48 ms 50 ms be6.cr2.syd7.on.ii.net [150.101.41.125]
9 50 ms 51 ms 49 ms be17.cr2.adl4.on.ii.net [150.101.33.33]
10 50 ms 50 ms 50 ms 150.101.40.146
11 47 ms 51 ms 51 ms mirror.internode.on.net [150.101.135.3]

Trace complete.

I have a similar path to @lorenceo from my home connection.

Tracing route to mirror.internode.on.net [150.101.135.3]
over a maximum of 30 hops:

1 1 ms <1 ms <1 ms thenet01.thenet.gen.nz [192.168.0.1]
2 4 ms 6 ms 18 ms 12.7.69.111.static.snap.net.nz [111.69.7.12]
3 * * * Request timed out.
4 120 ms 120 ms 121 ms 23655.syd.equinix.com [45.127.172.156]
5 125 ms 118 ms 119 ms 4739.syd.equinix.com [45.127.172.20]
6 143 ms 141 ms 142 ms be6.cr2.syd7.on.ii.net [150.101.41.125]
7 139 ms 138 ms 138 ms be17.cr2.adl4.on.ii.net [150.101.33.33]
8 142 ms 150 ms 146 ms 150.101.40.146
9 139 ms 138 ms 137 ms mirror.internode.on.net [150.101.135.3]

Trace complete.

https://nick.mackechnie.co.nz | NZ ISP latency monitoring - https://smokeping.thenet.gen.nz

JohnFlower

51 posts

Master Geek

#2617751 7-Dec-2020 19:42

Yup... Same here. :(

traceroute to mirror.internode.on.net (150.101.135.3), 30 hops max, 60 byte packets
1 oxalis.localdomain (192.168.1.1) 0.155 ms 0.123 ms 0.103 ms
2 108.7.69.111.static.snap.net.nz (111.69.7.108) 1.703 ms 1.681 ms 1.662 ms
3 * * *
4 23655.syd.equinix.com (45.127.172.156) 109.077 ms 109.058 ms 109.133 ms
5 4739.syd.equinix.com (45.127.172.20) 109.891 ms 109.873 ms 109.855 ms
6 be6.cr2.syd7.on.ii.net (150.101.41.125) 131.008 ms 131.406 ms 131.383 ms
7 be17.cr2.adl4.on.ii.net (150.101.33.33) 131.363 ms 131.235 ms 131.211 ms
8 * 150.101.40.146 (150.101.40.146) 128.537 ms *
9 * * *

Lorenceo

904 posts

Ultimate Geek

Trusted

#2618390 8-Dec-2020 23:22

Seems to be better now:

Tracing route to mirror.internode.on.net [150.101.135.3]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms ER4.lan [192.168.2.1]
2 3 ms 2 ms 4 ms 43.7.69.111.static.snap.net.nz [111.69.7.43]
3 * * * Request timed out.
4 25 ms 25 ms 25 ms 218.100.53.62
5 25 ms 25 ms 25 ms as4739.nsw.ix.asn.au [218.100.52.5]
6 47 ms 48 ms * be6.cr2.syd7.on.ii.net [150.101.41.125]
7 51 ms 50 ms 49 ms be17.cr2.adl4.on.ii.net [150.101.33.33]
8 45 ms 45 ms 45 ms 150.101.40.146
9 49 ms 49 ms 47 ms mirror.internode.on.net [150.101.135.3]

Tracing route to 4739.syd.equinix.com [45.127.172.20]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms ER4.lan [192.168.2.1]
2 2 ms 4 ms 3 ms 43.7.69.111.static.snap.net.nz [111.69.7.43]
3 * * * Request timed out.
4 25 ms 26 ms 27 ms 4739.syd.equinix.com [45.127.172.20]

Trace complete.

Tracing route to as56307.sydney.megaport.com [103.26.68.232]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms ER4.lan [192.168.2.1]
2 7 ms 3 ms 4 ms 43.7.69.111.static.snap.net.nz [111.69.7.43]
3 * * * Request timed out.
4 27 ms 27 ms 26 ms as56307.sydney.megaport.com [103.26.68.232]

Quic Broadband

Move to New Zealand's best fibre broadband service (affiliate link). Free setup code: R587125ERQ6VE. Note that to use Quic Broadband you must be comfortable with configuring your own router.

5thEntanglement

7 posts

Wannabe Geek

#2618567 9-Dec-2020 11:20

Hi, same problem here, weird route. AKL Central.

Sent 2degrees a short email ~3 weeks back.

Tracing route to mirror.internode.on.net [150.101.135.3]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms 192.168.1.1
2 2 ms 3 ms 3 ms 43.7.69.111.static.snap.net.nz [111.69.7.43]
3 * * * Request timed out.
4 26 ms 26 ms 26 ms 23655.syd.equinix.com [45.127.172.156]
5 162 ms 162 ms 162 ms 4739.syd.equinix.com [45.127.172.20]
6 182 ms 183 ms 183 ms be6.cr2.syd7.on.ii.net [150.101.41.125]
7 182 ms 183 ms 183 ms be17.cr2.adl4.on.ii.net [150.101.33.33]
8 182 ms 181 ms 181 ms 150.101.40.146
9 182 ms 181 ms 183 ms mirror.internode.on.net [150.101.135.3]

Trace complete.

pwner

423 posts

Ultimate Geek

Trusted

2degrees

#2618572 9-Dec-2020 11:23

5thEntanglement:

Hi, same problem here, weird route. AKL Central.

Sent 2degrees a short email ~3 weeks back.

Tracing route to mirror.internode.on.net [150.101.135.3]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms 192.168.1.1
2 2 ms 3 ms 3 ms 43.7.69.111.static.snap.net.nz [111.69.7.43]
3 * * * Request timed out.
4 26 ms 26 ms 26 ms 23655.syd.equinix.com [45.127.172.156]
5 162 ms 162 ms 162 ms 4739.syd.equinix.com [45.127.172.20]
6 182 ms 183 ms 183 ms be6.cr2.syd7.on.ii.net [150.101.41.125]
7 182 ms 183 ms 183 ms be17.cr2.adl4.on.ii.net [150.101.33.33]
8 182 ms 181 ms 181 ms 150.101.40.146
9 182 ms 181 ms 183 ms mirror.internode.on.net [150.101.135.3]

Trace complete.

are you still seeing this or has it been resolved as per the others above?

Any posts are personal comments and not that of my employer

5thEntanglement

7 posts

Wannabe Geek

#2618612 9-Dec-2020 12:36

@pwner

still the same issue for me. 9th Dec and counting.

A bunch of other servers give the same high latency, been like this for a while now.

Can be bypassed to some extent with a VPN. Brings ~300ms down to ~130ms

yitz

2083 posts

Uber Geek

#2618676 9-Dec-2020 14:22

What does the return path look like?

http://looking-glass.iinet.net.au select AS4739 Adelaide traceroute node and use your Internet IPv4 address as the target.

5thEntanglement

7 posts

Wannabe Geek

#2618726 9-Dec-2020 14:42

Not too sure what's going on. BE18-140.br02.hkg15.pccwbtn.net seems to be hong kong?

So perhaps I should request a new static ip from 2degrees>?

Router: AS4739 - Sydney, AU
Command: traceroute ip ***.***.***.***

Tracing the route to ***.***.***.***.static.2degrees.nz (***.***.***.***)

1 gi1-2-950.cor1.syd6.on.ii.net (150.101.199.50) 0 msec 0 msec 0 msec
2 te-0-9-1-3.cr2.syd7.on.ii.net (150.101.40.164) 0 msec 0 msec 0 msec
3 be51.cr2.syd2.on.ii.net (150.101.40.243) 0 msec 0 msec 0 msec
4 syd-apt-ros-wgw1-be-40.tpgi.com.au (202.7.162.249) 4 msec 0 msec 0 msec
5 syd-apt-ros-int2-be100.tpgi.com.au (203.29.134.4) 4 msec 4 msec
203.29.134-68.tpgi.com.au (203.29.134.68) 4 msec
6 BE18-140.br02.hkg15.pccwbtn.net (63.216.82.145) 144 msec 140 msec 140 msec
7 HundredGE0-6-0-2.br02.hkg12.pccwbtn.net (63.218.174.85) 144 msec 144 msec
TenGE0-1-0-17.br02.hkg12.pccwbtn.net (63.218.174.61) 144 msec
8 HundredGE0-6-0-2.br02.hkg12.pccwbtn.net (63.218.174.85) 144 msec
ae5.cr0-hkg2.ip4.gtt.net (69.174.124.169) 144 msec
HundredGE0-6-0-2.br02.hkg12.pccwbtn.net (63.218.174.85) 144 msec
9 xe-3-3-0.cr1-syd1.ip4.gtt.net (141.136.106.1) 244 msec
ae5.cr0-hkg2.ip4.gtt.net (69.174.124.169) 148 msec
xe-3-3-0.cr1-syd1.ip4.gtt.net (141.136.106.1) 248 msec
10 xe-3-3-0.cr1-syd1.ip4.gtt.net (141.136.106.1) 244 msec
ip4.gtt.net (67.199.137.170) 136 msec
xe-3-3-0.cr1-syd1.ip4.gtt.net (141.136.106.1) 240 msec
11 ip4.gtt.net (67.199.137.170) 140 msec 136 msec 140 msec
12 * * *
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *
18 * * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *
30 * * *

From AKL

Tracing route to gi1-2-950.cor1.syd6.on.ii.net [150.101.199.50]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms 192.168.1.1
2 2 ms 7 ms 3 ms 43.7.69.111.static.snap.net.nz [111.69.7.43]
3 * * * Request timed out.
4 25 ms 26 ms 26 ms 23655.syd.equinix.com [45.127.172.156]
5 162 ms 162 ms 162 ms 4739.syd.equinix.com [45.127.172.20]
6 163 ms 162 ms 162 ms be6.cr2.syd7.on.ii.net [150.101.41.125]
7 163 ms 162 ms 162 ms gi1-2-950.cor1.syd6.on.ii.net [150.101.199.50]

Trace complete.

yitz

2083 posts

Uber Geek

#2618791 9-Dec-2020 16:14

Looks to affect the routing prefixes below, long way around too with Optus.

Command: show ip bgp regex _3257_23655$

Network Next Hop Metric LocPrf Weight Path

*> 103.66.124.0/22 150.101.32.15 0 4739 7545 3491 3257 23655 i

*> 103.232.104.0/22 150.101.32.15 0 4739 7545 3491 3257 23655 i

* 103.66.124.0/22 203.202.143.34 0 7474 7473 1299 23655 i
*> 203.202.143.33 0 7474 7473 1299 23655 i
* 103.232.104.0/22 203.202.143.33 0 7474 7473 1299 23655 i
*> 203.202.143.34 0 7474 7473 1299 23655 i

aspired

32 posts

Geek

ID Verified

Trusted

2degrees

#2618857 9-Dec-2020 19:36

5thEntanglement:
Not too sure what's going on.

Can you please PM me your 2degrees username so I can look into this further

5thEntanglement

7 posts

Wannabe Geek

#2618921 9-Dec-2020 20:30

My main target is 72.5.161.228. Which is perma on a bad route

C:\WINDOWS\system32>tracert 72.5.161.228

Tracing route to 72.5.161.228 over a maximum of 30 hops

1 <1 ms <1 ms <1 ms 192.168.1.1
2 4 ms 10 ms 7 ms 43.7.69.111.static.snap.net.nz [111.69.7.43]
3 * * * Request timed out.
4 126 ms 126 ms 126 ms TWODEGREES-svc063403-ic344202.c.telia.net [62.115.175.227]
5 127 ms 126 ms 126 ms las-b24-link.telia.net [62.115.175.226]
6 127 ms 126 ms 126 ms singaporetelecom-ic-335365-las-b24.c.telia.net [62.115.8.203]
7 126 ms 126 ms 128 ms 203.208.171.117
8 289 ms 289 ms 289 ms 203.208.182.41
9 298 ms 297 ms 297 ms 203.208.182.253
10 298 ms 299 ms 299 ms 203.208.172.93
11 297 ms 297 ms 297 ms 203.208.177.10
12 297 ms 297 ms 295 ms border1.xe-0-1-1.bbnet1.sin001.pnap.net [202.58.8.8]
13 296 ms 295 ms 295 ms edge1.ae1-edgenet.sin003.pnap.net [202.58.9.14]
14 296 ms 295 ms 295 ms 202.58.10.198
15 296 ms 295 ms 295 ms 72.5.161.228

Trace complete.

With vpn:

C:\WINDOWS\system32>tracert 72.5.161.228
Tracing route to 72.5.161.228 over a maximum of 30 hops

1 25 ms 26 ms 26 ms 10.11.128.1
2 26 ms 26 ms 26 ms gi-43.es-18-513-p1.as45671.net.au [202.130.32.17]
3 27 ms 28 ms 28 ms vlif4084.cs-18.as45671.net.au [118.127.7.180]
4 26 ms 26 ms 28 ms 100.64.105.8
5 27 ms 38 ms 26 ms ve1001.br-1-2.as45671.net.au [203.26.198.15]
6 26 ms 26 ms 26 ms 100.64.120.125
7 28 ms 26 ms 26 ms 100.64.121.11
8 27 ms 25 ms 28 ms 202.60.93.236
9 118 ms 119 ms 117 ms hundredgige0-0-1-0.bdr01-ipt-47bourke-syd.au.superloop.net.co [103.200.13.229]
10 118 ms 120 ms 119 ms hundredgige0-0-1-2-128.bdr01-ipt-1william-per.au.superloop.net.co [103.200.13.189]
11 117 ms 117 ms 117 ms fortygige0-0-1-2-131.bdr01-ipt-26aayerr-sin.sg.superloop.net.co [202.177.40.20]
12 118 ms 119 ms 119 ms internap.sgix.sg [103.16.102.44]
13 117 ms 117 ms 120 ms bbr2.ae7.sin001.pnap.net [216.52.198.30]
14 120 ms 120 ms 119 ms tsr2.sin3.agile.pnap.net [216.52.198.50]
15 119 ms 119 ms 119 ms 72.26.212.154
16 119 ms 120 ms 119 ms 72.5.161.228

But the targets mentioned in the above posts are back to normal.

150.101.199.50 (iiNet AS4379 Internode)
150.101.135.3 (mirror.internode.on.net)

C:\Users\**>tracert 150.101.199.50

Tracing route to gi1-2-950.cor1.syd6.on.ii.net [150.101.199.50]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms 192.168.1.1
2 4 ms 3 ms 3 ms 43.7.69.111.static.snap.net.nz [111.69.7.43]
3 * * * Request timed out.
4 25 ms 26 ms 26 ms 23655.syd.equinix.com [45.127.172.156]
5 25 ms 26 ms 26 ms 4739.syd.equinix.com [45.127.172.20]
6 25 ms 26 ms 26 ms be6.cr2.syd7.on.ii.net [150.101.41.125]
7 48 ms 25 ms 26 ms gi1-2-950.cor1.syd6.on.ii.net [150.101.199.50]

Trace complete.

C:\Users\**>tracert 150.101.135.3

Tracing route to mirror.internode.on.net [150.101.135.3]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms 192.168.1.1
2 4 ms 3 ms 3 ms 43.7.69.111.static.snap.net.nz [111.69.7.43]
3 * * * Request timed out.
4 24 ms 26 ms 25 ms 23655.syd.equinix.com [45.127.172.156]
5 26 ms 26 ms 38 ms 4739.syd.equinix.com [45.127.172.20]
6 45 ms 46 ms 46 ms be6.cr2.syd7.on.ii.net [150.101.41.125]
7 47 ms 48 ms 48 ms be17.cr2.adl4.on.ii.net [150.101.33.33]
8 45 ms 46 ms 46 ms 150.101.40.146
9 45 ms 46 ms 46 ms mirror.internode.on.net [150.101.135.3]

Trace complete.

1 | 2