'Size' of Internet in NZ

Forums › New Zealand Broadband › 'Size' of Internet in NZ

1 | 2 | 3 | 4 | 5

637 posts

Ultimate Geek

Trusted

#417435 15-Dec-2010 08:27

ojala: Alexa lists TradeMe, NZ Herald and Stuff, followed by on-line banks, the largest local NZ sites. Take something like NZ Herald, what level of traffic do they generate? 1 Gbit/s at peaks?

I'd be surprised if it was anything near a gigabit for NZ Herald. 200Mbps at most, and more likely less than 100Mbps.

ojala: How much traffic do the ISP's exchange, does any of the NZ IX's publish their statistics? APE, WIX, at what level are the traffic through them, 10 Gbit/s, 20 Gbit/s? Do the largest IX'es already do 10GE?

Sadly no IXPs in NZ post statistics, largely due to their legacy architecture which makes it hard to identify IXP traffic vs. customer traffic.

Anecdotally I'd imagine the traffic at APE and WIX to each be 2Gbps or under; anecdotal evidence of other private peering links would suggest maybe another 1-2Gbps on top of that. The IXes do not (currently) do 10GE because there's no demand for it. The volume of traffic in New Zealand over all is very low, and the onshore traffic is extremely low.

ojala: Take a random big NZ ISP, how much international capacity do they have in place? Multiple 10GE's? A smaller ISP, GE with a few hundred Mbit/s of CIR?

There's a huge gulf between the largest ISP and the next largest. While the largest might be in significant tens of Gigabits, the next largest probably isn't. I'd be very surprised if any but the largest two were running "multiple 10GEs" that were actually carrying significant Gigabits. They might be a 10G port but sub-rate services.

freitasm: As part of our work I started using a CDN. I couldn't find anything locally, I guess because a New Zealand CDN would only be viable if a large provider could peer with Telecom, Vodafone and TelstraClear in each of the major locations (WLG, AKL, CHC, DUD), but seriously I don't think it would affect speed that much.

I guess you could talk to Orcon now...

Zeon: There was a site that was posted on GPforums which had some info about various AS (autonomous system numbers, ISPs usually would have one for their BGP peering). Ran a few ISPs. Not sure how accurate but:

This almost certainly would have been PeeringDB.com (Telecom, Orcon) but note that these are self-published indicators only (you could easily put "1Tbps" in there if you wanted) and you really need to understand what you're looking at with those statistics. For instance TNZ's includes their retail and wholesale customer bases, including those in Australia.

freitasm

BDFL - Memuneh
79250 posts

Uber Geek

Administrator

ID Verified

Trusted

Geekzone

Lifetime subscriber

#417439 15-Dec-2010 08:32

PenultimateHop:
freitasm: As part of our work I started using a CDN. I couldn't find anything locally, I guess because a New Zealand CDN would only be viable if a large provider could peer with Telecom, Vodafone and TelstraClear in each of the major locations (WLG, AKL, CHC, DUD), but seriously I don't think it would affect speed that much.

I guess you could talk to Orcon now...

Thanks, but no thanks.

wjw

162 posts

Master Geek

#417442 15-Dec-2010 08:42

PenultimateHop: Anecdotally I'd imagine the traffic at APE and WIX to each be 2Gbps or under; anecdotal evidence of other private peering links would suggest maybe another 1-2Gbps on top of that. The IXes do not (currently) do 10GE because there's no demand for it. The volume of traffic in New Zealand over all is very low, and the onshore traffic is extremely low.

I think you underestimate how much data goes through the exchanges. Just look at the number of ISP's using the Google Cache thats advertised via APE or TV OnDemand or Trademe etc etc etc.

Most of the main hosting providers in NZ will have connectivity at the IX's, it's far cheaper to peer yourself than buy domestic bandwidth from TNZ or A N Other ISP.

Of course the big two still don't have a presence at the IX's.

Cheers,

Bill

PenultimateHop

637 posts

Ultimate Geek

Trusted

#417444 15-Dec-2010 08:49

wjw: I think you underestimate how much data goes through the exchanges. Just look at the number of ISP's using the Google Cache thats advertised via APE or TV OnDemand or Trademe etc etc etc.

Possibly but unlikely. I've not heard anything concrete that proves it one way or another; but I have seen many port statistics at the IX and also CDN delivery statistics from a number of CDNs.

Not all CDN related (e.g. GGC) traffic traverses IX fabric either.

Edit to add: I'd love to be proven wrong on this, by the way.

wjw: Most of the main hosting providers in NZ will have connectivity at the IX's, it's far cheaper to peer yourself than buy domestic bandwidth from TNZ or A N Other ISP.

Of course the big two still don't have a presence at the IX's.

Absolutely - but most of the amounts are pretty trivial from what I've seen - it's all well and dandy having 160 IXP participants but if 150 of them are doing less than 1Mbps then the other 10 better be doing a huge amount of traffic!

As you point out the biggest two traffic sinks (and semi-sources) in NZ aren't at the IXPs which removes a huge amount of traffic from IXPs, and that changes the scenario significantly. I've seen some statistics from a reasonably large hosting provider which shows about as much traffic towards TNZ (only) as towards APE (as a whole).

PenultimateHop

637 posts

Ultimate Geek

Trusted

#417447 15-Dec-2010 08:52

freitasm: Thanks, but no thanks.

Well, they are a domestic CDN (apparently) which seems to be a better fit for your needs. Or Citylink might be able to help...

wjw

162 posts

Master Geek

#417449 15-Dec-2010 08:55

PenultimateHop: Edit to add: I'd love to be proven wrong on this, by the way.

I would love to prove you wrong, and from what I've seen you are, but due to commercial sensitivities I can't...

I should add that with certain CDN's their closest nodes are still in Sydney

freitasm

BDFL - Memuneh
79250 posts

Uber Geek

Administrator

ID Verified

Trusted

Geekzone

Lifetime subscriber

#417450 15-Dec-2010 08:56

PenultimateHop:
freitasm: Thanks, but no thanks.

Well, they are a domestic CDN (apparently) which seems to be a better fit for your needs. Or Citylink might be able to help...

We are currently using MaxCDN, because of their pull zone structure. Look at our pages and you will notice we already do a good job of optimizing images and resources (thanks to www.aptimize.com).

A few reasons: 60% of our traffic is from overseas, so having a CDN with POPs in US, Europe is a good thing. Cost is very good, compared to what could be done locally. And we don't have a traffic problem in New Zealand.

[EDIT TO ADD]: We still have our servers in Auckland with Datacom. Only static resources are served from the CDN.

Equinox IT

Cloud spending continues to surge globally, but most organisations haven’t made the changes necessary to maximise the value and cost-efficiency benefits of their cloud investments. Download the whitepaper From Overspend to Advantage now.

PenultimateHop

637 posts

Ultimate Geek

Trusted

#417453 15-Dec-2010 09:06

wjw:
PenultimateHop: Edit to add: I'd love to be proven wrong on this, by the way.

I would love to prove you wrong, and from what I've seen you are, but due to commercial sensitivities I can't...

That's unfortunate, as I'd really love to know the real answer to this question, much like ojala does.

Another way of looking at this: if New Zealand's overall international connectivity is 100Gbps (which anecdotally seems high), and it's commonly accepted (InternetNZ has reported on this) that there's a 9:1 ratio of international/domestic traffic, that would imply around 11Gbps of domestic traffic on-top of the 100Gbps. If 50% of that goes across peering (again, seems high) then that puts peering (IXPs and private interconnects) at 5-6Gbps, which fits within the model I proposed.

wjw: I should add that with certain CDN's their closest nodes are still in Sydney

Sure, although most of them have some kind of presence in NZ now, or if they're still Sydney based may be reaching IXP participants thanks to participants like Vocus. There's only one (maybe two) large CDNs that's missing from NZ and the story behind why is long and nasty.

freitasm: We are currently using MaxCDN, because of their pull zone structure. Look at our pages and you will notice we already do a good job of optimizing images and resources (thanks to www.aptimize.com).

A few reasons: 60% of our traffic is from overseas, so having a CDN with POPs in US, Europe is a good thing. Cost is very good, compared to what could be done locally. And we don't have a traffic problem in New Zealand.

[EDIT TO ADD]: We still have our servers in Auckland with Datacom. Only static resources are served from the CDN.

Sure, although as you point out 40% of your traffic is domestic and you're effectively forcing that traffic offshore (with the corresponding latency impact; and of course during the international outages making GZ render weird) to the nearest CDN node.

Although looking at MaxCDN's site I see they have plans for a Sydney node so that would improve on their coverage nicely.

Zeon

3916 posts

Uber Geek

Trusted

#417455 15-Dec-2010 09:06

PenultimateHop:
freitasm: Thanks, but no thanks.

Well, they are a domestic CDN (apparently) which seems to be a better fit for your needs. Or Citylink might be able to help...

Yea defiantely this - Freistasm, why not use a CDN who at least has a NZ node? It would be a big plus for many of us...!

Speedtest 2019-10-14

freitasm

BDFL - Memuneh
79250 posts

Uber Geek

Administrator

ID Verified

Trusted

Geekzone

Lifetime subscriber

#417456 15-Dec-2010 09:08

Cost, look at the cost. And reliability.

cyril7

9058 posts

Uber Geek

ID Verified

Trusted

Subscriber

#417464 15-Dec-2010 09:24

large CDNs that's missing from NZ and the story behind why is long and nasty

Keep talking, have never let a long and nasty story put me off before, unless of course its so far off topic that it should be taken elsewhere.

Cyril

wjw

162 posts

Master Geek

#417468 15-Dec-2010 09:35

cyril7:
large CDNs that's missing from NZ and the story behind why is long and nasty

Keep talking, have never let a long and nasty story put me off before, unless of course its so far off topic that it should be taken elsewhere.

Cyril

In the discussions I had about getting a Limelight node over here there just wasn't enough traffic. when a 12000 user ISP is only doing 500kbps aggregate and they require 1gbit to peer, I don't see anyone who fits that peering policy, at least no one other than maybe TNZ and TCL combined. But considering they share Akamai through your international routing table I don't see a LLNW node being freely available in the short to medium term.

PenultimateHop

637 posts

Ultimate Geek

Trusted

#417473 15-Dec-2010 09:43

wjw:
cyril7:
large CDNs that's missing from NZ and the story behind why is long and nasty

Keep talking, have never let a long and nasty story put me off before, unless of course its so far off topic that it should be taken elsewhere.

Cyril

In the discussions I had about getting a Limelight node over here there just wasn't enough traffic. when a 12000 user ISP is only doing 500kbps aggregate and they require 1gbit to peer, I don't see anyone who fits that peering policy, at least no one other than maybe TNZ and TCL combined. But considering they share Akamai through your international routing table I don't see a LLNW node being freely available in the short to medium term.

That's pretty much it in a nutshell (for LLNW anyway). Their architecture means they need extremely high bandwidth demand to make it viable to deploy a POP somewhere, and from their perspective they can reach almost all of New Zealand through existing peering connections in Australia or US West Coast so it's not worth it for them (unless someone wants to underwite the cost significantly).

The stats for traffic delivered into NZ are shockingly low although that could be a chicken-and-egg situation (low demand because of low bandwidth caps; low bandwidth caps because international transit is expensive... etc).

ojala

188 posts

Master Geek

#417570 15-Dec-2010 12:26

PenultimateHop:

Another way of looking at this: if New Zealand's overall international connectivity is 100Gbps (which anecdotally seems high), and it's commonly accepted (InternetNZ has reported on this) that there's a 9:1 ratio of international/domestic traffic, that would imply around 11Gbps of domestic traffic on-top of the 100Gbps. If 50% of that goes across peering (again, seems high) then that puts peering (IXPs and private interconnects) at 5-6Gbps, which fits within the model I proposed.

I brainstormed a bit with an old colleague and we estimated some rough figures for Finland.

A handful of major ISP's, the academics. adsl2+ has been available for years, HFC has been up to 200M for a few years now, FTTH is expanding pretty fast with up to 100M. Speedtest gives an average of 12.67 Mbit/s but we think the median is higher. 5.3 million people, slightly bigger area than NZ, very similar city populations.

We estimated the amount of international traffic around 200 Gbit/s, the norm is to have double the capacity actually needed (takes care of both redundancy and growth). Domestic traffic is around 100 Gbit/s, a third through the IX'es and rest through private peerings and other similar arrangements. That would be 2:1 ratio for internat/domestic traffic. The largest content sites go up to 1.5-2 Gbit/s but there's some heavy traffic inside the ISP networks due IPTV, PVR's in the cloud and other similar services (not included here). A single network of ~6000 student residences generate more than 1.5 Gbit/s alone.

Domestic bandwidth costs a few eurocents (~0.04 NZD) per Mbps, international about 5 euros (9 NZD) per Mbps. Global CDN and cache nodes exist and Google is building a big datacenter (mostly for Russia, though) but ISP's don't proxy/cache traffic (*. Private peering is attractive because it's cheap and it's flexible -- IX is a shared fabric after all.

The reason for 2:1 ratio is two fold, language is one thing but most people follow international sites as well. The other, big, reason is that as data caps have never existed during the broadband era. This has enabled the domestic content to grow freely since day 1.

*) Piece of history.. when www was just Tim's work at CERN, http proxy wasn't invented, and bandwidth was scarce we established nic.funet.fi, one of the largest FTP sites at the time. We mirrored a number of FTP sites from US, brought content closer and turned Finland into the first and at the time the only country in Europe who was actually putting out more traffic than in.

PenultimateHop

637 posts

Ultimate Geek

Trusted

#417583 15-Dec-2010 12:43

ojala: A handful of major ISP's, the academics. adsl2+ has been available for years, HFC has been up to 200M for a few years now, FTTH is expanding pretty fast with up to 100M. Speedtest gives an average of 12.67 Mbit/s but we think the median is higher. 5.3 million people, slightly bigger area than NZ, very similar city populations.

We estimated the amount of international traffic around 200 Gbit/s, the norm is to have double the capacity actually needed (takes care of both redundancy and growth). Domestic traffic is around 100 Gbit/s, a third through the IX'es and rest through private peerings and other similar arrangements. That would be 2:1 ratio for internat/domestic traffic. The largest content sites go up to 1.5-2 Gbit/s but there's some heavy traffic inside the ISP networks due IPTV, PVR's in the cloud and other similar services (not included here). A single network of ~6000 student residences generate more than 1.5 Gbit/s alone.

Domestic bandwidth costs a few eurocents (~0.04 NZD) per Mbps, international about 5 euros (9 NZD) per Mbps. Global CDN and cache nodes exist and Google is building a big datacenter (mostly for Russia, though) but ISP's don't proxy/cache traffic (*. Private peering is attractive because it's cheap and it's flexible -- IX is a shared fabric after all.

The reason for 2:1 ratio is two fold, language is one thing but most people follow international sites as well. The other, big, reason is that as data caps have never existed during the broadband era. This has enabled the domestic content to grow freely since day 1.

All sounds entirely valid to me. Sadly New Zealand is nowhere near those cost figures or bandwidth figures, and I wouldn't put it entirely down to data-caps (after all, Australia has data caps too and their traffic volumes are markedly different). Language definitely does have a contributing factor to domestic traffic exchange in my view, but inconsistently. India is an interesting but maybe invalid example here due to the subregional languages.

ojala: *) Piece of history.. when www was just Tim's work at CERN, http proxy wasn't invented, and bandwidth was scarce we established nic.funet.fi, one of the largest FTP sites at the time. We mirrored a number of FTP sites from US, brought content closer and turned Finland into the first and at the time the only country in Europe who was actually putting out more traffic than in.

I remember downloading things from nic.funet.fi mirrors back in the mid-90s. Takes me back to an entirely different era of shonky satellite connectivity and imuxed E1s.

1 | 2 | 3 | 4 | 5