Unable to access websites I manage unless using VPN/Proxy. Spark network blocking?

Forums › Spark New Zealand (including Skinny and BigPipe) › Unable to access websites I manage unless using VPN/Proxy. Spark network blocking?

1 | 2 | 3

955 posts

Ultimate Geek

Trusted

Spark NZ

#3051267 17-Mar-2023 17:23

pureinception:

inspectaclueso:

Sorry to hear you are having the same issue but glad it is not just me. It's driving me crazy going from one support person to the next from both sides and no one seeming to know what is going on or taking any responsibility.

Yeah I hear you, It's very frustrating! It started on Tuesday and had been down since. Neither party will accept any responsibility. Namecheap even suggested switching providers while Spark suggests using a VPN. I wish they could work together and fix the root cause.

The issue here is that the problem is not in the Spark network. Instead it is an issue with a provider across the other side of the world that Spark has no relationship (commercial or otherwise). The issue has been highlighted to that provider as per the reply above. Unfortunately beyond that we have no way of forcing that provider to fix their issue.

My views are my own, and may not necessarily represent those of my employer.

yitz

2083 posts

Uber Geek

#3051274 17-Mar-2023 17:46

cbrpilot: The issue here is that the problem is not in the Spark network. Instead it is an issue with a provider across the other side of the world that Spark has no relationship (commercial or otherwise). The issue has been highlighted to that provider as per the reply above. Unfortunately beyond that we have no way of forcing that provider to fix their issue.

So you are confident there is no BGP route hijacking in Spark's Domestic routing table going on here?

inspectaclueso

29 posts

Geek

#3051275 17-Mar-2023 17:49

Getting dizzy from all the piggy in the middle.

yitz

2083 posts

Uber Geek

#3051277 17-Mar-2023 17:52

That "voyager-dom.akcr11.global-gateway.net.nz [122.56.118.162]" hop is on a Voyager router, specifically edge01.mdr.vygr.net

You could give Voyager's NOC contact a try https://www.nznog.org/noc-list

cbrpilot

955 posts

Ultimate Geek

Trusted

Spark NZ

#3051280 17-Mar-2023 18:09

yitz:

cbrpilot: The issue here is that the problem is not in the Spark network. Instead it is an issue with a provider across the other side of the world that Spark has no relationship (commercial or otherwise). The issue has been highlighted to that provider as per the reply above. Unfortunately beyond that we have no way of forcing that provider to fix their issue.

So you are confident there is no BGP route hijacking in Spark's Domestic routing table going on here?

Here is the ASpath of that route:

4648 (Global Gateway)

56030 (Voyager)

137409 (GSL Networks)

12189 (PhoenixNAP LLC)

20454 (Target Training Intl)

22612 (Namecheap)

I'm not going to claim expertise in regards to route validation etc (not my area), but it doesn't look obviously bad to my untrained eye.

My views are my own, and may not necessarily represent those of my employer.

cbrpilot

955 posts

Ultimate Geek

Trusted

Spark NZ

#3051281 17-Mar-2023 18:14

inspectaclueso:

Getting dizzy from all the piggy in the middle.

It would appear they are basing that opinion purely based on the tracetcp result. Show them a full traceroute that shows the packet getting all the day to the destination please.

As per my previous message it would be informative for all parties to know if the TCP-syn packets are getting to your webserver, and if it is replying. Then at least you'd know if the issue was with getting to the destination, or getting back again.

My views are my own, and may not necessarily represent those of my employer.

docjolly

2 posts

Wannabe Geek

#3051386 17-Mar-2023 23:12

I'm having the same issue.

No access to privateemail.com (namecheap's mail servers) from Spark network (fibre and cellular), fine via VPN, and fine from myRepublic.

And 'sorry not our problem' responses from both Spark and Namecheap support channels (though do appreciate input above cbrpilot).

😡 so annoying it inspired my 1st post on geekzone...

Quic Broadband

Move to New Zealand's best fibre broadband service (affiliate link). Free setup code: R587125ERQ6VE. Note that to use Quic Broadband you must be comfortable with configuring your own router.

pureinception

21 posts

Geek

#3051394 17-Mar-2023 23:43

docjolly:

I'm having the same issue.

No access to privateemail.com (namecheap's mail servers) from Spark network (fibre and cellular), fine via VPN, and fine from myRepublic.

And 'sorry not our problem' responses from both Spark and Namecheap support channels (though do appreciate input above cbrpilot).

😡 so annoying it inspired my 1st post on geekzone...

My first time on here too because of this frustrating issue. I've been back and fourth with support with no progress. I tried ping and nslookup from the server to the spark.co.nz IP:

Unsure what to do to resolve the email issue 🤨

yitz

2083 posts

Uber Geek

#3051400 17-Mar-2023 23:55

cbrpilot:

Here is the ASpath of that route:

4648 (Global Gateway)

56030 (Voyager)

137409 (GSL Networks)

12189 (PhoenixNAP LLC)

20454 (Target Training Intl)

22612 (Namecheap)

I'm not going to claim expertise in regards to route validation etc (not my area), but it doesn't look obviously bad to my untrained eye.

It seems this is a legitimate but complicated routing setup (going by the complaints probably only a few days old) so you are seeing the 198.187.28.0/22 from the Domestic route table but there is also a 198.187.29.0/24 on the International route table and it seems Spark will always prefer the entry from Domestic table even if less specific which is different to how it works on Vodafone (who also have a Domestic/International split) who are seeming to use the 198.187.29.0/24 from Telstra Global. Eventually it comes to back to Voyager how this has been designed as they are presumably paid by GSL Networks to pick up this traffic from Spark customers.

cbrpilot

955 posts

Ultimate Geek

Trusted

Spark NZ

#3051402 18-Mar-2023 00:18

yitz:

It seems this is a legitimate but complicated routing setup (going by the complaints probably only a few days old) so you are seeing the 198.187.28.0/22 from the Domestic route table but there is also a 198.187.29.0/24 on the International route table and it seems Spark will always prefer the entry from Domestic table even if less specific which is different to how it works on Vodafone (who also have a Domestic/International split) who are seeming to use the 198.187.29.0/24 from Telstra Global. Eventually it comes to back to Voyager how this has been designed as they are presumably paid by GSL Networks to pick up this traffic from Spark customers.

That is very interesting info Yitz, appreciated. I've passed that onto Global Gateway for them to investigate.

My views are my own, and may not necessarily represent those of my employer.

yitz

2083 posts

Uber Geek

#3051403 18-Mar-2023 00:20

There's also a DDoS scrubbing/mitigation provider in the path so best not to be sending too many TCP SYN packets 😃

nic.wise

333 posts

Ultimate Geek

Trusted

#3051427 18-Mar-2023 09:17

No solution but a friend is having the same issue with the same combination - spark fibre and mobile to a namecheap hosted mail and web server.

Nic Wise - fastchicken.co.nz

yitz

2083 posts

Uber Geek

#3051443 18-Mar-2023 11:06

I see routing has changed so has access been restored for OP and others?

docjolly

2 posts

Wannabe Geek

#3051446 18-Mar-2023 11:11

yitz:

I see routing has changed so has access been restored for OP and others?

Working for me now from Spark mobile network.

Firebirdnz

35 posts

Geek

Trusted

Voyager

#3051448 18-Mar-2023 11:13

Hi folks of GeekZone,

The official @VygrNetworkMonkey is on holiday at the moment. I'm stepping in in his place. Firstly I'd like to apologise for the trouble that has been caused by those here (and others) around this issue. We believe this issue is now resolved (hopefully we will have some confirmation in some replies here).

Now to the meat of this. How did this situation occur?

Issues were caused by the following factors:

End ASN inconsistently deaggregating prefixes - For example they advertised a /22 one way and a /24 another way
Spark Domestic not having full routing knowledge of what is in Spark GGI
Voyager employing Anti-Spoof / BCP38 filters

If one of these factors was removed the traffic would have passed.

From the Voyager perspective we were receiving the /22 prefix from our downstream peering relationship with GSL. We were also receiving the /24 prefixes from our upstream peering relationship with Spark GGI.

On the return path for networks connected to Spark Domestic our prefix was the best. It would route to us via our peering with Spark domestic, we would complete a route lookup see the more specific /24 and route it back to Spark GGI. From our networks perspective the source IP was something on Spark and the destination was Namecheap. Our Anti-Spoof filters blocked this traffic flow. We only allow either Voyager owned prefixes or prefixes advertised to us by our downstream customers to exit our network.

The issues with Namecheap unfortunately were not isolated. On Tuesday we completed a large prefix update and a number of those prefixes in that update were subject to similar issues. Some of those were observed by Spark customers, others by those on peering exchanges in New Zealand. For those on peering exchanges the commonality would have been that they did not have full global routing knowledge and only saw our less specific advertisement.

We were first made aware of issues on Thursday and we began working to try and remove one of the above listed factors. This was not a simple task however, more to that below. We have instead opted to employ some smarts to the prefixes we receive from our external non customer peering relationships (IP Transit, IX's etc.) We are now blocking any more specific prefix for any prefix we receive from a downstream customer from coming from one of those other external peerings. This will ensure that any inconsistent advertisement from the source ASN is ignored and we will route direct to a customer.

We have opted to do this line of work because we cannot control the routing policies and advertisement decisions of another ASN. We cannot get them to be consistent. We also cannot control the network architecture spark employ. We also don't have control over the other third parties that may opt to not run full routing tables. We could have removed our Anti-Spoof filters but we believe these are important for any network operator in order to limit participation in DDOS's. Further to this, without these filters, in this case, Spark would have in essence been getting free transit from us. These filters also limit the ability for Exchange peers or other non transit peered networks to get free capacity from us.

So in short, we apologise for dropping packets, we have made changes, hopefully all is well now, and I hope you can understand how this situation came about.

~ Another Voyager Network Monkey...

1 | 2 | 3