Spark peering

Forums › Spark New Zealand (including Skinny and BigPipe) › Spark peering

1 | 2 | 3 | 4 | 5 | 6 | 7 | 8

2566 posts

Uber Geek

#2107120 13-Oct-2018 08:20

If you use syncthing in NZ, the local public (based in auckland, peered at akl-ix and others) syncthing relay no longer allows spark IPs due to the lack of peering. (They're dropped at the firewall). This is because the relay was starting to push a lot of traffic via non-IX peering. (That is, via paid transit) (Mostly to spark customers).

If this affects you, complain to spark about their lack of open/settlement free peering. :)

muppet

2572 posts

Uber Geek

Trusted

#2107121 13-Oct-2018 08:27

kyhwana2: If you use syncthing in NZ, the local public (based in auckland, peered at akl-ix and others) syncthing relay no longer allows spark IPs due to the lack of peering. (They're dropped at the firewall). This is because the relay was starting to push a lot of traffic via non-IX peering. (That is, via paid transit) (Mostly to spark customers).

If this affects you, complain to spark about their lack of open/settlement free peering. :)

I love Syncthing, but this must affect what, 6 people? :)

BarTender

3607 posts

Uber Geek

ID Verified

Trusted

Lifetime subscriber

#2107123 13-Oct-2018 08:35

muppet:

freitasm:

Time to access/download resources - there's a bit of difference between AKL and Narita in terms of the distance bits have to travel...

Right, but it doesn't present an actual problem, aside from longer RTT's and slightly longer page load time. Which for a HTTP2 page is unlikely to be noticed by most users, it's not exactly taking seconds to load (is it?)

Don't get me wrong, I agree the lack of local in-country traffic routing seems silly. Back when I worked at one of the large telco's (not Spark) and we (network engineering) were told this de-peering had to take place and that we had to turn off the APE peers, we argued vehemently against it. The senior people asked us "But what will actually break?" and we had to, begrudgingly, admit that nothing would break, some traffic might route a slightly longer way etc. (We were also told if we didn't do it and kept moaning about, we'd be shown the door, so we turned it down as instructed)

Anyway, I don't think it's a problem. Slightly slower RTT, sure.

Finally, there's always two sides to the peering story. I'm not saying Spark aren't the "bad guys" here, but you also don't know for sure that Spark are holding out signing some contract because it exposes them in some way that the other ISPs have just rolled over and signed. [That is meant only as an example, not based in any factual information at all!!]

I agree with much of what was said above, knowing how some of the inner working worked back when I was at Spark there were all sorts of issues with smaller ISPs taking the p..s that has lead to the situation today.

The main issue I see is more links with smaller players means more BGP import and export filters to manage and operational cost overhead with managing additional capacity links and ensuring that no one fat-fingers a routes.

The Spark core is incredibly simple and designed to be that way as shifting 8+PB a day and having a peak of over 500GBit/s back when I was there. More complexity = more cost = more operational management = harder to get changes deployed into production as everything had to be tested.

This when costs are continuing to go down in regards to monthly plan charges vs the cost of delivering a network where capacity doubles every year.

raytaylor: I am pretty sure that you can buy spark peering in different cities, and buy a backhaul product to those cities.
Spark basically know they have a high number of customers and if you want your mutual customers to have a better experience with your online service, you need to pay spark to deliver the traffic from a closer source.

Spark don't understand who their customer is.
However rugby world cup may change the dynamics of how spark peers... I still have hope.

Jessie from @fullflavor has some experience with getting to spark customers via Australia

The RWC content will all be delivered from Akamai CDN nodes installed within the ISPs network so it will never leave the network so have no material impact on peering. I think the control serves are hosted in AWS but that is just for authorising the streams and is a tiny amount of traffic vs the actual stream. What I can suspect will happen is smaller ISPs who don't have Akamai peering will struggle, and other ISPs who have under invested in their own Akamai CDN nodes will also struggle. That to me should be a far larger concern for Spark as they have no control over Vodafone or 2D etc on how much investment they put into installing more Akamai CDNs to cover the anticipated load. That being said I would bet coming closer to the date there will be plenty of ad's from Spark "Sparks network is scaled to handle the load for RWC but your current ISP isn't, come join us and get all these free things.."

While I personally don't agree with not peering, I know it will add a whole lot of extra complexity onto engineers that are already busy enough keeping up with current changes.

freitasm

BDFL - Memuneh
79316 posts

Uber Geek

Administrator

ID Verified

Trusted

Geekzone

Lifetime subscriber

#2112811 23-Oct-2018 15:32

Unrelated but related... Cloudflare peering portal: option 3 would work for Spark if they don't want to peer directly:

Scenario 3: Hosting a Node

There are scenarios where neither PNI nor IxP peering is feasible, such as when Cloudflare and our potential partners are not present in the same physical location. In these cases, our partner can request to host Cloudflare equipment in their own data center racks. Once commissioned, the equipment effectively operates as if it were a Cloudflare data center (PoP). The cache will populate and all of our performance and reliability services will execute right there, close to our peering partner's customer.

michaelmurfy

meow
13271 posts

Uber Geek

Moderator

ID Verified

Trusted

Lifetime subscriber

#2112818 23-Oct-2018 16:13

freitasm:

Unrelated but related... Cloudflare peering portal: option 3 would work for Spark if they don't want to peer directly:

Scenario 3: Hosting a Node

There are scenarios where neither PNI nor IxP peering is feasible, such as when Cloudflare and our potential partners are not present in the same physical location. In these cases, our partner can request to host Cloudflare equipment in their own data center racks. Once commissioned, the equipment effectively operates as if it were a Cloudflare data center (PoP). The cache will populate and all of our performance and reliability services will execute right there, close to our peering partner's customer.

This is the problems - the reason is politics, not that they are not present. They could peer if they wanted to but instead they simply don't.

Which is why my traffic to Cloudflare currently is going via "Tokyo, Japan (NRT)" on a corporate connection via Spark where my home connection is taking a more direct route just 5 hops away served by the APE.

Not to mention it actually "feels" slow to browse Geekzone on a Spark connection no, that's no placebo, it is fact looking at the waterfall.

Just excellent. It is nice that traffic takes a scenic route over to Japan and back to NZ where Geekzone is hosted.

Michael Murphy | https://murfy.nz
Referral Links: Quic Broadband (use R122101E7CV7Q for free setup)

Are you happy with what you get from Geekzone? Please consider supporting us by subscribing.
Opinions are my own and not the views of my employer.

Sounddude

I fix stuff!
1928 posts

Uber Geek

Trusted

2degrees

Lifetime subscriber

#2112882 23-Oct-2018 20:06

BarTender:

I agree with much of what was said above, knowing how some of the inner working worked back when I was at Spark there were all sorts of issues with smaller ISPs taking the p..s that has lead to the situation today.

The main issue I see is more links with smaller players means more BGP import and export filters to manage and operational cost overhead with managing additional capacity links and ensuring that no one fat-fingers a routes.

The Spark core is incredibly simple and designed to be that way as shifting 8+PB a day and having a peak of over 500GBit/s back when I was there. More complexity = more cost = more operational management = harder to get changes deployed into production as everything had to be tested.

This when costs are continuing to go down in regards to monthly plan charges vs the cost of delivering a network where capacity doubles every year.

As a large network operator (who has worked for pretty much everyone ISP in NZ now), I find that quite hard to believe. Most Large operators use BGP communities these days to manage their export rules and input rules consist of "stop anything stupid from happening". They are very low touch.

The main reason comes down to commercials. Spark don't want to give up the hundred of thousands dollars or so they get from ISP's and business who have to pay for domestic traffic to reach spark with low latency. This is why spark always say "No business case to peer" and this is why. They risk losing a lot of revenue.

Its the same thing over in Australia with Telstra and Optus and several Tier1 providers around the world. (Level3 etc).

I don't agree with it, but I understand why.

sbiddle

30853 posts

Uber Geek

Retired Mod

Trusted

Biddle Corp

Lifetime subscriber

#2113028 24-Oct-2018 07:20

Spark's lack of peering will ultimately be one of the downfalls of their RWC coverage and push into the TV space. I see a change of policy before the RWC - not because Spark want to, but because they simply have no other option.

I'm pretty confident Spark can built a streaming platform to handle what is required, and that their own users will have a great experience. That doesn't mean every other RSP will, especially those having to interconnect in Australia.

The average end user doesn't care about politics and $$ - they just want their Internet to work, and when it doesn't work during the RWC there will be riots in the street with pitchforks.

Quic Broadband

Move to New Zealand's best fibre broadband service (affiliate link). Free setup code: R587125ERQ6VE. Note that to use Quic Broadband you must be comfortable with configuring your own router.

Talkiet

4793 posts

Uber Geek

Trusted

#2113055 24-Oct-2018 08:47

I strongly suspect that investment in CDNs will be a much bigger predictor of Rugby World Cup success than significant changes to our peering policies.

I've seen the rwc argument a couple of times on here now being put forward by people without any knowledge of how the event will be provisioned to spark and other customers and thought it was time to correct this misunderstanding.

Cheers N

Please note all comments are from my own brain and don't necessarily represent the position or opinions of my employer, previous employers, colleagues, friends or pets.

Sounddude

I fix stuff!
1928 posts

Uber Geek

Trusted

2degrees

Lifetime subscriber

#2113125 24-Oct-2018 09:49

Slightly offtopic

but AU are going though the same thing and the ACCC have deemed the Gang of 3 of not being anti-competitive with their peering stance aslong as they make public the criteria required for peering.

(Worth noting that Telstra is now peering with Vocus).

https://www.computerworld.com.au/article/648621/no-need-regulate-isp-peering-arrangements-accc-says/

michaelmurfy

meow
13271 posts

Uber Geek

Moderator

ID Verified

Trusted

Lifetime subscriber

#2116417 29-Oct-2018 17:18

Hmm - I've been annoyed for quite some time on the status of Spark peering but this is another reason where it can negatively affect the customer and an good example:

I personally run Apache Guacamole inside a Docker container however as a layer of security this is run via Cloudflare. This is also hosted in NZ (on my home server). On most ISP's it is totally fine - 14ms latency, SSH / RDP / VNC works totally fine however on Spark this goes via Tokyo at a 500ms round trip which makes SSH quite slow and RDP / VNC unusable. It was actually quicker to use this via my mobile, tethering (2degrees - 40ms RTT) than via the Spark Fibre connection.

Cloudflare isn't just for websites - actual services can be hosted behind it also. I think it is somewhat important for ISP's to peer to such things given more and more websites, app servers and also API's are sitting behind Cloudflare these days.

So, in response to the following:

muppet:

freitasm: Time to access/download resources - there's a bit of difference between AKL and Narita in terms of the distance bits have to travel...

Right, but it doesn't present an actual problem, aside from longer RTT's and slightly longer page load time. Which for a HTTP2 page is unlikely to be noticed by most users, it's not exactly taking seconds to load (is it?)

Don't get me wrong, I agree the lack of local in-country traffic routing seems silly. Back when I worked at one of the large telco's (not Spark) and we (network engineering) were told this de-peering had to take place and that we had to turn off the APE peers, we argued vehemently against it. The senior people asked us "But what will actually break?" and we had to, begrudgingly, admit that nothing would break, some traffic might route a slightly longer way etc. (We were also told if we didn't do it and kept moaning about, we'd be shown the door, so we turned it down as instructed)

Anyway, I don't think it's a problem. Slightly slower RTT, sure.

Finally, there's always two sides to the peering story. I'm not saying Spark aren't the "bad guys" here, but you also don't know for sure that Spark are holding out signing some contract because it exposes them in some way that the other ISPs have just rolled over and signed. [That is meant only as an example, not based in any factual information at all!!]

I give you the following:

Yes, it does affect more than just websites. It can affect apps that work behind Cloudflare (API's, Websockets (in my case) and other things). Cloudflare isn't just for websites and moreso since they're expanding into other areas now. But, peering will also assist services that go via Spark (eg - a particular large bank or 2 where some traffic has to go via Australia now to reach their apps).

yitz

2083 posts

Uber Geek

#2116427 29-Oct-2018 17:29

The connection on I'm currently is on an ISP which supposedly peer at the Auckland exchanges and I'm also being sent to Tokyo for most of Cloudflare for the past... nearly a month?

Have to agree can definitely notice the delays.

michaelmurfy

meow
13271 posts

Uber Geek

Moderator

ID Verified

Trusted

Lifetime subscriber

#2172135 3-Feb-2019 11:28

I noticed that Geekzone (and another site I often visit) was terribly slow. Just put it down to the website and the ADSL connection I am on currently. But, it looks like this has gotten worse:

Location
Hong Kong, Hong Kong (HKG)

mmurphy ~ ping geekzone.co.nz
PING geekzone.co.nz (104.24.3.14): 56 data bytes
64 bytes from 104.24.3.14: icmp_seq=0 ttl=55 time=173.815 ms
64 bytes from 104.24.3.14: icmp_seq=1 ttl=55 time=178.946 ms
64 bytes from 104.24.3.14: icmp_seq=2 ttl=55 time=178.910 ms
64 bytes from 104.24.3.14: icmp_seq=3 ttl=55 time=172.855 ms
^C
--- geekzone.co.nz ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 172.855/176.131/178.946/2.817 ms

If I fire up NordVPN (using the NZ node) I get a respectable 14ms ping to Cloudflare websites and a connection via Auckland. Sites are far snappier.

The route even goes via Sydney:

mmurphy ~ traceroute geekzone.co.nz
traceroute: Warning: geekzone.co.nz has multiple addresses; using 104.24.2.14
traceroute to geekzone.co.nz (104.24.2.14), 64 hops max, 52 byte packets
1 192.168.0.1 (192.168.0.1) 6.129 ms 6.858 ms 7.430 ms
2 * * *
3 * * *
4 ae8-10.akbr6.global-gateway.net.nz (122.56.116.5) 17.309 ms 17.892 ms 19.361 ms
5 ae2-6.tkbr12.global-gateway.net.nz (122.56.127.17) 19.610 ms
ae7-2.akbr7.global-gateway.net.nz (122.56.119.53) 19.010 ms
ae2-6.tkbr12.global-gateway.net.nz (122.56.127.17) 20.755 ms
6 xe0-0-7.sebr3.global-gateway.net.nz (202.50.232.6) 39.557 ms
xe5-0-0.sgbr3.global-gateway.net.nz (202.50.232.242) 44.555 ms
xe0-0-4.sgbr3.global-gateway.net.nz (122.56.127.94) 43.620 ms
7 ae7-10.sebr4.global-gateway.net.nz (122.56.127.214) 43.729 ms
122.56.119.86 (122.56.119.86) 46.103 ms 42.862 ms
8 i-0-4-0-0.sydp01.bi.telstraglobal.net (202.84.223.10) 45.828 ms
122.56.119.86 (122.56.119.86) 43.490 ms
i-0-4-0-0.sydp01.bi.telstraglobal.net (202.84.223.10) 49.516 ms
9 i-0-1-0-7.sydp-core03.bi.telstraglobal.net (202.84.223.9) 45.822 ms 46.401 ms 45.968 ms
10 i-11101.hkhh-core02.telstraglobal.net (202.84.138.45) 157.939 ms 162.041 ms 160.955 ms
11 i-0-2-0-2-1.hkgg01.telstraglobal.net (202.84.157.173) 158.837 ms
i-0-5-0-3-0.hkgg01.telstraglobal.net (202.84.157.166) 165.107 ms
i-0-2-0-3-7.hkgg01.telstraglobal.net (202.84.157.189) 160.846 ms
12 unknown.telstraglobal.net (210.57.81.22) 164.226 ms 165.077 ms 163.466 ms
13 104.24.2.14 (104.24.2.14) 158.320 ms 159.059 ms 157.886 ms

Talkiet

4793 posts

Uber Geek

Trusted

#2172136 3-Feb-2019 11:32

about 6 days ago Cloudflare started serving some traffic to Spark customers from HongKong instead of from Aussie. I don't know why - it's up to them or their customers.

I have had a look at the level of traffic coming into AS4771 from Cloudflare in the last couple of days and unfortunately it's still a rounding error so doesn't justify any investment from an international traffic saving point of view.

There would be a benefit from a user experience point of view, but in most cases the improvement would be virtually imperceptible (only about 15% of the CF traffic flicked over to HongKong - the majority is still Sydney)

Cheers - N

Please note all comments are from my own brain and don't necessarily represent the position or opinions of my employer, previous employers, colleagues, friends or pets.

michaelmurfy

meow
13271 posts

Uber Geek

Moderator

ID Verified

Trusted

Lifetime subscriber

#2172137 3-Feb-2019 11:40

@Talkiet Cheers - I'm finding 100% of the sites I visit that are on Cloudflare are served out of HK or Japan currently. They do however route via Australia which is potentially what you're seeing. The "Cloudflare Pro / Enterprise" sites I've got are going via HK and the Free sites are going via Japan. I'm yet to see one served out of Australia, but this may be just because of the sites I visit.

Talkiet

4793 posts

Uber Geek

Trusted

#2172140 3-Feb-2019 11:47

michaelmurfy:

@Talkiet Cheers - I'm finding 100% of the sites I visit that are on Cloudflare are served out of HK or Japan currently. They do however route via Australia which is potentially what you're seeing. The "Cloudflare Pro / Enterprise" sites I've got are going via HK and the Free sites are going via Japan. I'm yet to see one served out of Australia, but this may be just because of the sites I visit.

Nope - I am logged into a cloudflare portal and I can see about 20% of traffic now at peak times is coming from Hong Kong, about about 60-70% from Sydney and the remainder from Japan and other places. (This is just for AS4771 traffic)

Cheers - N

Please note all comments are from my own brain and don't necessarily represent the position or opinions of my employer, previous employers, colleagues, friends or pets.

1 | 2 | 3 | 4 | 5 | 6 | 7 | 8