2D (& DTS) issue with AWS sites/services???

Forums › 2degrees (including Slingshot, Orcon, Flip, Stuff Fibre, MyRepublic, 2talk and Vocus) › 2D (& DTS) issue with AWS sites/services???

mobiusnz

457 posts

Ultimate Geek

#310989 5-Dec-2023 13:10

Had a client ring to say he couldn't get to https://connect.protel.net (A property management service site) this morning. He's on DTS and I'm on 2D so suggested he go back to Protel.

Later had a client who's copiers couldn't login to their billing system with printix - https://sign-in.printix.net/ - He's 2D as am I, I did a DNS lookup and noticed its AWS as is the Protel.net one.

I have confirmed both sites work perfectly on Spark connections and Vodafone connections but don't work on 2D or DTS.

I've called both companies - DTS have escalated it but they can get to them from their head office - They are going to have an engineer test it in CHCH.

The 2D tech confirmed he couldn't access them at all either so he is trying to talk to a manager to get the issue escalated and he'll ring me back.

Anyone else seen anything like this today? Having two SAS pages not working both hosted on AWS makes me thing there are likely others.

Hopefully one or both will get to the bottom of the issue, hopefully its not an Upstream Issue that will go in a queue with their supplier but we'll see.

Matt Beechey Mobius Network Solutions

1 | 2

4085 posts

Uber Geek

ID Verified

Trusted

#3168317 5-Dec-2023 13:19

Interesting seeing this post, been trying to troubleshoot sign-on issues to Dashlane this morning where the SSO starts on an AWS site.

NickMack

962 posts

Ultimate Geek

Trusted

Lifetime subscriber

#3168322 5-Dec-2023 13:27

Hiya,

From my 2d home fibre connection

tracert connect.protel.net

Tracing route to public-ireland-cloud-ssl-605067142.eu-west-1.elb.amazonaws.com [52.17.190.184]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms thenet01.thenet.gen.nz [192.168.0.1]
2 2 ms 1 ms 1 ms 12.7.69.111.static.snap.net.nz [111.69.7.12]
3 * * * Request timed out.
4 16 ms 16 ms 22 ms default-rdns.vocus.co.nz [101.98.5.231]
5 16 ms 16 ms 16 ms default-rdns.vocus.co.nz [101.98.5.230]
6 16 ms 17 ms 16 ms as9790.bdr01.akl03.akl.nz.vocus.network [175.45.102.238]
7 16 ms 17 ms 16 ms be1000.bdr01.akl03.akl.nz.vocus.network [175.45.102.237]
8 * * * Request timed out.
9 131 ms * * be202.lsr01.dody.nsw.vocus.network [103.1.77.14]
10 131 ms 131 ms * be803.lsr01.prth.wa.vocus.network [103.1.76.147]
11 131 ms 131 ms 131 ms be200.cor01.per04.wa.vocus.network [103.1.77.113]
12 136 ms 136 ms 137 ms be201.bdr01.sin01.sin.vocus.network [114.31.206.51]
13 136 ms 135 ms 136 ms 80.81.65.217
14 327 ms 328 ms 328 ms 62.216.129.182
15 295 ms 283 ms 279 ms 62.216.131.153
16 * * * Request timed out.
17 * * * Request timed out.
18 * * * Request timed out.
19 * * * Request timed out.

From My home Spark wireless connection

tracert connect.protel.net

Tracing route to public-ireland-cloud-ssl-605067142.eu-west-1.elb.amazonaws.com [52.17.190.184]
over a maximum of 30 hops:

1 8 ms 1 ms 1 ms SmartModem2.thenet.gen.nz [192.168.1.254]
2 433 ms 49 ms 58 ms 10.207.150.169
3 * * * Request timed out.
4 76 ms 66 ms 74 ms 122.56.119.216
5 100 ms 77 ms 78 ms ae10-10.tkbr12.global-gateway.net.nz [202.50.232.29]
6 197 ms 373 ms 262 ms xe11-0-2.lebr7.global-gateway.net.nz [122.56.119.126]
7 236 ms 509 ms 280 ms ae3-10.sjbr3.global-gateway.net.nz [122.56.127.25]
8 268 ms 293 ms 304 ms ae0.pabr5.global-gateway.net.nz [203.96.120.74]
9 325 ms 249 ms 335 ms palo-b24-link.ip.twelve99.net [62.115.145.204]
10 435 ms 343 ms 283 ms nyk-bb1-link.ip.twelve99.net [62.115.118.120]
11 * * * Request timed out.
12 534 ms 403 ms 583 ms ldn-b2-link.ip.twelve99.net [62.115.120.239]
13 * * * Request timed out.
14 * * * Request timed out.
15 * * * Request timed out.
16 * * * Request timed out.
17 * ^C

Works fine, quite different paths.

Looks like the server is located in AWS Ireland - Are there ranges that need to be whitelisted to allow connections, the following might help.

IP ranges for AS23655 - https://ipinfo.io/AS23655 (2degrees)

IP ranges for AS9790 - https://ipinfo.io/AS9790 (Vocus)

Nick

https://nick.mackechnie.co.nz | NZ ISP latency monitoring - https://smokeping.thenet.gen.nz

mobiusnz

457 posts

Ultimate Geek

#3168323 5-Dec-2023 13:28

DjShadow:

Interesting seeing this post, been trying to troubleshoot sign-on issues to Dashlane this morning where the SSO starts on an AWS site.

I've just forwarded some tracert's to 2D but as he said - Its not going to be quick isolating where the issue is.

Strangely I've also found it works on my 2D mobile if I turn Wifi off so its different on 2D for Mobile / Fibre broadband??

Matt Beechey Mobius Network Solutions

NickMack

962 posts

Ultimate Geek

Trusted

Lifetime subscriber

#3168324 5-Dec-2023 13:30

mobiusnz:

DjShadow:

Interesting seeing this post, been trying to troubleshoot sign-on issues to Dashlane this morning where the SSO starts on an AWS site.

I've just forwarded some tracert's to 2D but as he said - Its not going to be quick isolating where the issue is.

Strangely I've also found it works on my 2D mobile if I turn Wifi off so its different on 2D for Mobile / Fibre broadband??

Yep - Different IP Ranges :-)

https://nick.mackechnie.co.nz | NZ ISP latency monitoring - https://smokeping.thenet.gen.nz

mobiusnz

457 posts

Ultimate Geek

#3168369 5-Dec-2023 14:50

NickMack:

Hiya,

From my 2d home fibre connection

tracert connect.protel.net

Tracing route to public-ireland-cloud-ssl-605067142.eu-west-1.elb.amazonaws.com [52.17.190.184]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms thenet01.thenet.gen.nz [192.168.0.1]
2 2 ms 1 ms 1 ms 12.7.69.111.static.snap.net.nz [111.69.7.12]
3 * * * Request timed out.
4 16 ms 16 ms 22 ms default-rdns.vocus.co.nz [101.98.5.231]
5 16 ms 16 ms 16 ms default-rdns.vocus.co.nz [101.98.5.230]
6 16 ms 17 ms 16 ms as9790.bdr01.akl03.akl.nz.vocus.network [175.45.102.238]
7 16 ms 17 ms 16 ms be1000.bdr01.akl03.akl.nz.vocus.network [175.45.102.237]
8 * * * Request timed out.
9 131 ms * * be202.lsr01.dody.nsw.vocus.network [103.1.77.14]
10 131 ms 131 ms * be803.lsr01.prth.wa.vocus.network [103.1.76.147]
11 131 ms 131 ms 131 ms be200.cor01.per04.wa.vocus.network [103.1.77.113]
12 136 ms 136 ms 137 ms be201.bdr01.sin01.sin.vocus.network [114.31.206.51]
13 136 ms 135 ms 136 ms 80.81.65.217
14 327 ms 328 ms 328 ms 62.216.129.182
15 295 ms 283 ms 279 ms 62.216.131.153
16 * * * Request timed out.
17 * * * Request timed out.
18 * * * Request timed out.
19 * * * Request timed out.

Looks like the server is located in AWS Ireland - Are there ranges that need to be whitelisted to allow connections, the following might help.

IP ranges for AS23655 - https://ipinfo.io/AS23655 (2degrees)

IP ranges for AS9790 - https://ipinfo.io/AS9790 (Vocus)

Nick

Interesting that you can get there on your 2D fibre when the helpdesk people can't ?? I get the exact same for a tracert but can't get to the site still - Nothing restricting IP's etc (EDIT: Same routes but different IP resolved - Whether thats just load balancing - Whose DNS are you using?)

C:\Users\Matt>tracert connect.protel.net

Tracing route to public-ireland-cloud-ssl-605067142.eu-west-1.elb.amazonaws.com [34.253.29.96]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms 192.168.69.1
2 2 ms 2 ms 2 ms v3.cpchn-hn-bng1.tranzpeer.net [101.98.0.122]
3 14 ms 14 ms 14 ms 10.101.0.158
4 15 ms 15 ms 15 ms as9790.bdr01.akl03.akl.nz.vocus.network [175.45.102.238]
5 15 ms 15 ms 15 ms be1000.bdr01.akl03.akl.nz.vocus.network [175.45.102.237]
6 * * * Request timed out.
7 * * * Request timed out.
8 * 130 ms * be803.lsr01.prth.wa.vocus.network [103.1.76.147]
9 129 ms 129 ms 130 ms be200.cor01.per04.wa.vocus.network [103.1.77.113]
10 134 ms 134 ms 134 ms be201.bdr01.sin01.sin.vocus.network [114.31.206.51]
11 151 ms 130 ms 130 ms 80.81.65.217
12 357 ms 321 ms 321 ms 62.216.129.182
13 290 ms 290 ms 288 ms 62.216.131.153
14 * * * Request timed out.
15 * * * Request timed out.
16 * * * Request timed out.
17 * * * Request timed out.
18 * * * Request timed out.

Also the other URL on a Tracert on 2D does go to twelve99.net which connect.protel.net does when on Spark so they must both be on AWS Ireland.

Matt Beechey Mobius Network Solutions

yitz

2075 posts

Uber Geek

#3168381 5-Dec-2023 15:20

I've also noticed AWS eu-west-1 (Ireland) and eu-west-2 (London) not reachable today in Christchurch from behind Compass (who use Voyager who in turn use Vocus routes for Europe).

http://ec2-reachability.amazonaws.com/

Has to be some filtering on the return path as forward path is going via Spark Global-Gateway but return is via Orange (France Telecom) mutual upstream of those AWS regions and Vocus 4826. ICMP/ping passes through fine.

https://www.globaltraceroute.com/

rugrat

3107 posts

Uber Geek

Lifetime subscriber

#3168431 5-Dec-2023 15:31

Works through VPN, but browser window just hangs on 2Degrees fibre with no VPN.

Looks like IP address range being blocked?, as different DNS doesn’t fix either.

Sharesies

Trade NZ and US shares and funds with Sharesies (affiliate link).

mobiusnz

457 posts

Ultimate Geek

#3168432 5-Dec-2023 15:38

yitz:

I've also noticed AWS eu-west-1 (Ireland) and eu-west-2 (London) not reachable today in Christchurch from behind Compass (who use Voyager who in turn use Vocus routes for Europe).

http://ec2-reachability.amazonaws.com/

Has to be some filtering on the return path as forward path is going via Spark Global-Gateway but return is via Orange (France Telecom) mutual upstream of those AWS regions and Vocus 4826. ICMP/ping passes through fine.

https://www.globaltraceroute.com/

Thanks Yitz - That ec2-reachability tool is brilliant - I went to the AWS network status pages etc and everything was good as the issue isn't there end. This will hopefully provide some info but it appears the problem is well upstream of 2D etc so its probably going to be a long wait only for the issue to go away due to someone who tripped over a cable plugging it back in 😂

Real PITA for my client who can't use their PMS for a Hotel - The other is a collegue who has issues with a billing system for a photocopier for billing clients for prints/copies.

Matt Beechey Mobius Network Solutions

andysh

228 posts

Master Geek

#3168443 5-Dec-2023 16:13

I am also having issues reaching a corporate system that is AWS EU based. That EC2 tool is having no luck accessing eu-west-1 and eu-west-2.

All ok if I go through a VPN.

Referrals:

Tesla: https://ts.la/andrew897313

Sharesies: https://sharesies.com/r/XRGS77

mobiusnz

457 posts

Ultimate Geek

#3168445 5-Dec-2023 16:15

Bit of an update for all - 2 Degrees still haven't come back to me but a DTS engineer has provided a workaround for my client who couldn't access their PMS (Protel.net) hosted on AWS in Ireland.

They have changed the clients MTU to 1382 as they found it was a packet size issue somewhere with an upstream provider.

I have to give the prize to DTS as I'm fully aware that it gets tricky when the issue might be more than one provider upstream of you but they provided a valid workaround.

So if anyone is having the issue try dropping the MTU on your router!

EDIT: I tried it on my Router (a Mikrotik) - Dropped the MTU size on the PPPoE connecton and it dropped and reconnected and the sites worked. So if the issue is bigger than one PC you can VPN from that'll provide a work around.

Matt Beechey Mobius Network Solutions

andysh

228 posts

Master Geek

#3168542 5-Dec-2023 19:08

This has now been resolved for me (I did nothing).

Referrals:

Tesla: https://ts.la/andrew897313

Sharesies: https://sharesies.com/r/XRGS77

mobiusnz

457 posts

Ultimate Geek

#3168545 5-Dec-2023 19:11

andysh:

This has now been resolved for me (I did nothing).

just tried myself and it’s going here too (I’m DTS at home and 2D at the office)

Matt Beechey Mobius Network Solutions

FailedWOF

45 posts

Geek

#3168561 5-Dec-2023 20:16

Also knocked out a few things for me. Working now though. Traceroutes show Vocus has rerouted traffic to AWS eu-west through AU and Asia instead of the US. Latency is horrible, but at least it works.

The problem looks like it's in Arelions connections to eu-west from Paris (Telehouse DC). That's where the traces stoped at least (but their Looking Glass showed a few more hops inside AWS).

OneNZ and Devoli were OK. They were routing through Telstra Global and Cogent respectively. Quic was also green on the eu-west EC2 connectivity check as well but don't know what path their traffic took.

mobiusnz

457 posts

Ultimate Geek

#3169760 8-Dec-2023 14:44

And 3 days later 2 Degrees have finally escalated my fault log to someone who has emailed and asked be to ring the call centre and push 9 to speak to anyone at the helpdesk right back at the beginning even though the person on the day confirmed they also couldn't access the sites in question.

Top Job 2 Degrees.

Matt Beechey Mobius Network Solutions

richms

28172 posts

Uber Geek

Trusted

Lifetime subscriber

#3170085 9-Dec-2023 16:36

Would this be affecting accessing xero from a slingshot connection? Don't want to waste time looking into it if it's part of the same outage

Richard rich.ms

1 | 2