It's The Muppet Show!


Understanding Traffic Flow of a NZ ISP.

, posted: 1-Nov-2010 17:00

Generic ISP NZ Limited

This post attempts to describe how Internet traffic with a fairly standard ISP works in New Zealand, from the point of a fictional ISP called “Generic ISP NZ Limited”.  I can’t stress enough that this isn’t modelled on any actual ISP in New Zealand!

The first thing to note is if you work in an ISP in New Zealand is that I've taken a few creative liberties with a few things.  Feel free to point these out in the comments and I’ll tell you if I meant to do that, or I’ll correct the article as required.  This is a fairly generic model I'm discussing here, I’m also talking mostly about ADSL services here too, though most things are relevant regardless of access method.

I'm also not talking much about the portion of the network from your house to the DSLAM.  There are other more knowledgeable people than me that can describe how it all works in detail.

Ok, so welcome to Generic ISP New Zealand Ltd!  We’re based at 101 Fictional Street in Central Auckland.

Let's look at our core network.  This is where all our traffic has come in from our ADSL/Cable/3G/Dedicated links, across our backhaul connections and now has to route to the final destination.  We now have a few different places we could send the traffic.

Here's Generic ISP NZ’s Network Diagram (click it for bigger):

 
Generic ISP NZ's Network Diagram
 

Traffic Links
Generic ISP NZ has 3 distinct different traffic paths out of the network.  The following section discusses these paths, where they go and the costs associated with them.

International Traffic
The first sort of traffic link we have to buy is International traffic.  Our users would be pretty unhappy if they couldn't load Facebook, Twitter and JustinBeiber.com (Why God? Why?)

All of our user’s traffic that has to “leave the country” travels over this link.

This links is very expensive, so we only buy 100Mb/s of capacity.  This is despite the fact we actually have a Gigabit Ethernet connection to our upstream, they rate limit us to 100Mb/s because that’s all we’re paying for.

During peak hours, this link tends to run flat out and some of our users see a little bit of packet loss.  The loss isn’t bad however and it’s only for 2-3 hours per night, so we’re sticking with it for the moment.  We can’t afford to buy 130Mb/s like we’d need to keep the link uncongested and keep our subscriber prices the same currently.

Peering  Traffic
The next traffic link we have is our Peering traffic.  This is traffic to other ISPs in New Zealand towards sites like TradeMe, local FTP mirrors etc.  This is much cheaper than International traffic because most of the traffic is swapped over “free” peering sessions between the ISPs.

Peering is great because it benefits both parties; therefore neither party changes the other.  If we send all our traffic to Bob ISP Ltd for free, then they will send us all their traffic for free.  Because of this, the more traffic we can get to go over our peering links the better for us, because it’s taking traffic that’d otherwise go over other links.

The major peering exchange in New Zealand is called APE, the Auckland Peering Exchange.  ISPs connect into APE and are able to freely exchange traffic.

WIX, the Wellington Internet eXchange is another local peering point for ISPs that are in Wellington.  There are other peering exchanges in New Zealand.

We can send/receive as much traffic over our Gigabit Ethernet link to APE as we want; all we have to pay for is the physical connectivity to get to APE because it’s hosted in the SkyTower.

Another small but interesting thing to note is that when you peer at APE, you don’t actually have to peer with every single ISP at APE.  All you do is peer with the CityLink Hosted Route-Servers and they teach everyone everyone else’s routes.  CityLink do a really great job of hosting this service, cheers CityLink!

Domestic Traffic
There are two major ISPs in New Zealand that don't peer with anyone else, TelstraClear and Telecom New Zealand.  They do peer with each other though.  What this means for us is that we have to purchase a connection from either TelstraClear or Telecom in order to get access to the resources and traffic within their networks.  This cost costs us a fair amount of money, but it’s not as expensive like our International traffic is.  So we can afford to keep this uncongested at all times, but we operate it with a tight margin.

If we didn’t buy this link, we could still get to TelstraClear and Telecom, but we’d have to send traffic over our International link to get there.  That’s an expensive thing for us to do and our smart users would complain about it, so we’re forced into buying a link from TelstraClear or Telecom.

A lot of people would say this is a very dirty tactic from TelstraClear and Telecom, forcing us smaller players into buying a connection from them.  I’ll leave that discussion for another time.

Edit: The other side to this argument is that it's not fair for Telecom or TelstraClear to be offering the large amount of content within their network to a small ISP, when that small ISP has very little to offer in return.  Or the fact that Telecom/TelstraClear might have to backhaul traffic up from ChristChurch for example to hand it over at peering point in Auckland, if the smaller ISP peers only in Auckland.
Peering is a complex beast and there's the technical advantages and the commerical advantages.  I didn't present the commercial aspect, but it's important to note that it's there.  Personally I think peering is a good thing, but a beancounter would see it otherwise!

Some other ISPs in the market might not peer directly at APE like we do, instead they might just buy a single Domestic connection from TelstraClear or Telecom which contains all of TelstraClear’s, Telecom’s and APE’s routes in it.  Doing this means they only have to manage two links, an International one and a Domestic one.

We find it easier to have 3 links though we have a good idea of where our traffic is flowing.

Caching
Because International bandwidth is so much more expensive for us, our user's HTTP traffic over this link is subject to caching.  When we do it properly, caching is not noticeable to our users and in fact is beneficial to them as content served from the cache is much faster than content served from overseas.  It also saves us from having to transmit/receive data over our International link, freeing it up for other user traffic.

So caching is actually a good thing, despite what a lot of people think.  The reason that most people have the misconception that caching is bad is because when it's not working properly, stale or even just wrong content can be served.  This is pretty frustrating for our end users .  A lot of sites also don't provide proper information to our caches, causing us even more problems with stale content.  We usually have to bypass these sites as we learn about them, so that we don’t attempt to cache any of their content.

SpeedTest
A lot of people use the SpeedTest.net site as a measure of their ISPs speed and hopefully the above explanation will now help explain why it alone isn’t a great measure.  The reason it’s not is because SpeedTest traffic travels over the Peering (or Domestic) connection that your ISP has.  As discussed, these links are easy to keep upgraded and uncongested.  International links, where most of your day-to-day browsing and downloading will come from, are expensive and ISPs like ourselves experience congestion over it during peak hours.

So it’s possible for Generic NZ Ltd customers to a good SpeedTest result, but for them to still have a sub-optimal browsing experience during peak hours.

SpeedTest is a good indicator of your access method’s (ADSL/Mobile etc) maximum performance capability, not so much overall Internet experience you’ll actually get. 
Edit: It's also good to show you if your ISP is experiencing congestion over their backhaul links during peak hours (also quite common) as you'll get poor SpeedTest results showing this. [Thanks kyhwana2]

Summary
I hope this gives a good overview of how an ISP in New Zealand routes traffic around.

I am happy to answer any technical type questions or even post another blog with more detail if you're interested in this.

Cheers,
Muppet

PS: I'm crap with HTML/CSS so apologies for the bland look and horrid formatting.


Tag(s):           


Other related posts:
Install Your Own Mini Google
Traceroute








Comment by nate, on 1-Nov-2010 18:06

Good post, very interesting for someone like me who has no idea


Comment by freitasm, on 1-Nov-2010 18:11

Well done, very good and informative!


Comment by kyhwana2, on 1-Nov-2010 18:18

The speedtest will tell you if the backhaul from the dslam is congested won't it? Assuming the link from isp to test server isn't congested. I can only do 2-4mbit to xnets own speedtest server onpeak vs 13ish mbit off peak..


Author's note by muppet, on 1-Nov-2010 18:21

@ kyhwana2 Indeed it will.  Thanks for pointing out that oversight, I'll correct it now.


Comment by johnr, on 1-Nov-2010 19:01

@Tim do you have www.justinbeiber.com in your favorites by any chance? Good write up by the way!


Comment by jjnz1, on 1-Nov-2010 20:48

You've done a very good job of explaining this. Very interesting to read too!


Comment by michaelmurfy, on 1-Nov-2010 23:15

Good job! This sums up things quite nicely.


Comment by David, Terso IT, on 2-Nov-2010 00:41

Cheers! Enjoying reading over the Auckland Peering Exchange link now.


Comment by stuzzo, on 2-Nov-2010 10:43

Thanks for an informative article. Often surprised at lack of caching on international news sites front pages eg BBC News is a very slow loader as are many US sites (Slingshot).


Comment by Stu Fleming, on 2-Nov-2010 11:00

Nice diagram and explanation.
Disclaimer: I own and run WIC, a wireless ISP in Dunedin. My flow diagram looks somewhat different.

Some notes:
- Telecom NZ do actually have a local peering policy via the 29 regional peering points. They also interconnect via ICONZ and APE.
- TCL stands alone as you note
- there are 7 neutral peering points through the country under Citylink's ExchangeNet program. These are the well-known ones such as APE & WIX, but also include Hamilton, Palmerston North, 3 Cities, Christchurch and Dunedin. Only one ISP (FX Networks) peers at all of these (and with Telecom NZ)
- each exchange has a different character - APE is the national meeting point for many ISPs and large organizations; WIX does double-duty as an ISP meeting point and promotes local connectivity in Wellington; CHIX does the same function in ChCh
- Akamai have a local CDN (Content Distribution Node) connected directly via APE, which helps enormously for content from (among others) Microsoft and Apple, without an ISP necessarily having to do their own caching
- some ISPs e.g. telecom NZ, TCL, Orcon, Slingshot operate YouTube/Google caches to remove access to that content from their international links.
- Google's nearest data centre is Sydney, which is about 40-50ms latency and on international. Some international provision e.g. via Vocus can get to/from Oz at lower cost than via LAX.

- small correction when you say your user traffic "leaves the country", you really mean "enters the country". The majority of NZ ISPs would have a traffic profile 6:1 or 10: of download (inbound) over upload (outbound). My profile as a symmetric provider is 3:1 or even 1:1 on a good day.

As an independent ISP, I don't buy backhaul or international capacity from either Telecom NZ or TCL and I'm fortunate not to have any legacy ADSL access layer.

Future scalability:
The little icon for "cache farm" is actually going to be the killer for "conventional" ISPs when data rates go up. Your note on not being able to afford 130Mbps to avoid congestion resonates with me as I am in a similar situation. And there's probably a YouTube cache in there and maybe a torrent cache as well if you do an O**** Sandvine implementation.

But that presupposes the majority of your overseas traffic is inbound. And it presupposes that the majority of it is Web based and/or can be cached. Because that's a kind of short-term optimisation to wring the best interactive performance out of the international pipe. And if traffic patterns or the nature of content changes, that falls flat. I don't have a solution for this and I don't manage traffic this way, just making the note.

- the ratio of inbound to outbound traffic is also a killer. If we suggest that NZ consumers will suddenly become producers, that traffic needs to get oerseas. In an ADSL type setup, your upload rate is restricted by the technology. I'm a symmetric provider, so I unthrottle my upload pipes and get content out of the country as fast as possible. There's another implication there for ISPs to evolve to be able to manage large amounts of outbound traffic when upload speeds become faster under UFB.


Author's note by muppet, on 2-Nov-2010 11:35

Hi Stu,

Some really great comments, thank you!

Ol' Generic ISP NZ Ltd is missing a lot of detail that most ISPs have in their network, purely for simplicities sake.  Backhaul is a whole article in itself, as is peering policy, private peering, multi-homing, load balancing, routing policy etc.  All of which I think is a bit too specific for the Geekzone audience.

I'm aware of the local peering policy that Telecom have at their exchanges, yes.  What you may or may not realise is you're only getting access to their DSL customers, nothing more at the moment.  I believe there's plans to change that though and it's certainly a great step in the right direction.  There is some other noise of change in the peering world as well, which is nice!

Caching is a two sided sword. When it works it works well, but when it breaks... Keeping on top of that, especially when most sites are so dynamic these days is a battle I suspect.  It's really a cost benefit thing, is it cheaper to install and manage a cache farm? Or is it just cheaper to buy the extra bandwidth?  I don't have any experience with the P2P type caches that people use these days, I'd love to know how well they do (or don't) work.

Anyway thank you very much for your comment.  Good luck with WIC!

Tim


Author's note by muppet, on 2-Nov-2010 12:35

After a bit of offline discussion, I've updated my article to include the following around peering/domestic traffic.

Edit: The other side to this argument is that it's not fair for Telecom or TelstraClear to be offering the large amount of content within their network to a small ISP, when that small ISP has very little to offer in return.  Or the fact that Telecom/TelstraClear might have to backhaul traffic up from ChristChurch for example to hand it over at peering point in Auckland, if the smaller ISP peers only in Auckland.
Peering is a complex beast and there's the technical advantages and the commerical advantages.  I didn't present the commercial aspect, but it's important to note that it's there.  Personally I think peering is a good thing, but a beancounter would see it otherwise!


Comment by ojala, on 4-Nov-2010 08:01

Interesting to read..  and well in line what I figured out so far.

What is the cost relation between international bandwidth and bandwidth to Telecom NZ / TelstraClear?  Can you get a GE access to their networks for a decent price?

How is the bandwidth around the country, can you get GE or 10GE to all the top 3 or top 7 (>=100,000 people) cities for a decent price?

What is the average and prime time traffic through APE?

I'm just trying to figure out why the national traffic is still being included in the data cap (at full price).

Considering the population of NZ, I think the regulation should push towards "mandatory" peering in Auckland, Christchurch and Wellington.  That would cover a great portion of the population already.

When we had somewhat similar situation in the mid 90's, the rules for our national IX was that you needed to have your own international capacity to join the iX.

PS. To change the in:out ratio ADSL isn't a bit deal, most of the national traffic would become from other sources -- companies with symmetric connections, sites (web, blog, video streaming, ...) from hosted facilities with "unlimited" bandwidth, etc.  But before that happens, national traffic must become free.  The lack of free national traffic is one reason why the pressure is on the international bandwidth.


Author's note by muppet, on 4-Nov-2010 09:44

Hi Ojala,

Thanks for your comment.

I'd say at the moment it's about 3-4 times more expensive for International than it is for Domestic.  Prices depend a lot on how long you lock yourself into a contract for etc.  So really, trying to easily compare the two is quite difficult.
The physical port speed you connect up at doesn't really change the cost much.  It's the "Mb/s" you buy that has the cost attached to it.  You can be plugged into Telecom at 10Gb/s but you might still only be buying 20Mb/s over that connection.

The majority of my experience is only with APE and WIX, so I don't know what sort of prices you can get joined up to the others for.

Traffic "through APE" is also hard if not impossible to answer, unless you got a graph of each ISP's port there's no easy way to tell.  You could ask CityLink but again as that's commerically sensitive information, I don't think anyone will tell you.

The reason that National traffic is still included in the Data Cap is simple: Backhaul.  You might have a huge pipe up to APE but if all your backhaul is hitting its limits, the last thing you want is every house out there streaming "On Demand" 24/7.  You have to think of every aspect of the network, not just individual points.
It's been discussed before why the whole "National Traffic Free" idea is a bit of a falsehood.  You always end up paying somewhere.

Not sure I agree on mandatory peering.  How do you write policy for that?  Does that mean an ISP with 2 users and 512k International upstream can peer with TelstraClear and Telecom for free?
It's an interesting idea though.

Thanks for taking the time to write up your comment.

Tim


Comment by icepicknz, on 4-Nov-2010 13:50

Hey Ojala,

With regards to your costing on bandwidth, international has been getting cheaper and cheaper over the years where telecom/telstraclear have not dropped their domestic prices since the de-peering of the exchanges many years back.

The company I buy connectivity off sell CIR (commited information rate) at around the following, obviously it all depens on how much you buy of each:

1Mb CIR domestic (telecom/telstraclear) - $150
1mb CIR international - $220
APE - Free

You can however go into the burst pool, where you pretty much get 100mbps domestic most of the time while others aren't using it, but the above price for domestic is what you pay for CIR.

So as you can see domestic and international prices are very close. When you buying say 40mb of international and you're getting it at a price of $175 or something, then it becomes very close to equal that of domestic, then lets say you were buying 500mb of international for $100 a mb, whats the point in buying a few mb of domestic, just buy a few more mb of international as it's cheaper. Obviously this suggestion is not a good idea as the provider you sending the traffic to may only have a small international pipe, so the restriction wouldn't be on your end but rather the far end.

With regards to circuits metro and inter-metro, you looking at around $2400 for a 100mb circuit from one place in auckland to another, in wgtn, this is much cheaper because citylink offer much better pricing. Inter-metro pricing from say auckland to chch, of 100mb, you looking in the 10's of thousands of dollars unless you have huge capacity already with telstra clear or telecom.

What ever way you look at it, the more you buy, the cheaper it starts to become per mb or per 100mb.

Cheers
Barry Murphy

disclaimer: I contract to ISP's and telco's so what has been said is my opinion and mine only.


Comment by BlakJak, on 4-Nov-2010 16:48

For purposes of accuracy, Stu Fleming made the observation "Telecom NZ do actually have a local peering policy via the 29 regional peering points. They also interconnect via ICONZ and APE." 

ICONZ peer with Telecom (ala they pay Telecom for a circuit for purposes of transit).  So more accurately it should say 'They also interconnect with ICONZ' the same as i'm sure the majority of ISP's do; as you can see from this excellent article and diagram, if you wish to get into Telecom and TelstraClear's networks on behalf of your customers (or theirs), you need to pay them for the privilege.

Speaking personally, of course.

BJ.


Comment by wjw, on 5-Nov-2010 09:54

Saying you are buying 1mbit CIR International isn't exactly correct, an ISP can only ever guarantee that bandwidth to the edge of their network.

Also as far as splitting national/international traffic, The proportion of national traffic when averaged across an ISP's customer base is usually only around 10%-20%, this makes the commercial cost of splitting this traffic for volume based billing not viable.


Author's note by muppet, on 5-Nov-2010 10:01

@wjw: Don't understand what you mean.  That's how it's sold, the rate card will be 1Mb/s = $150 Month.

So it's as correct as you can be.  I don't think I understand your comment.


Comment by SuperT, on 5-Nov-2010 16:54

How about adding a bit of information as to how this traffic control affects gamers as a low latency is essential and quite a few ISP's will give problems for gamers at peak times.
It would be interesting to know which ISP treat gamers equitably?


Author's note by muppet, on 8-Nov-2010 06:05

Hi SuperT,

Thanks for your comment.  There is nothing specific to the way that traffic flows that will affect gaming in any way that wouldn't also affect other traffic.  If your traffic has to go Internationally then you'll get more latency than if the game server is local (over a Domestic or Peering connection)

The things that will affect you most, such a ISPs throttling gaming traffic, ADSL interleaving etc I haven't discussed here.

There is also specifically no mention of individual ISPs, so if you wish to discuss that the Geekzone forums would be a better option.  This article is intentionally ISP neutral.

Tim


Comment by wjw, on 8-Nov-2010 12:56

@muppet So it's as correct as you can be.  I don't think I understand your comment.

You are buying 1mbit to the edge of their upstream providers network, which is not how must people see it. So a Committed Information Rate doesn't actually apply to the 'internet'.

It's a common mis-interpretation that if you are buying say 10mbits to the internet you should be getting 10mbits to everywhere on the internet.


Author's note by muppet, on 8-Nov-2010 13:54

@ wjw: I'm talking about buying bandwidth wholesale from the likes of Vocus/Reach etc.  Is that what you're talking about?

I don't think I've suggested anywhere that buying 10Mb/s from your upstream will give you 10Mb/s to anywhere on the Internet.  You're buying a pipe size - it just means you can fit a lot more litte dribbles of data through it.

Or are you just talking about the fact that icepicknz has used the term CIR?  All that means in this instance is that you're not contending/sharing with anyone else.  The 10Mb/s is "all yours", it's not shared with any other ISP.

Of course there might potentially be congestion in the upstream's network, so it's not a pure 10Mb/s CIR, but for the purpose used I think the term's clear enough.


Comment by ChillingSilence, on 9-Nov-2010 14:53

A brilliant article!
There's so much mis-information out there, such as that by Chris Barton from NZHerald, where all he wants to do is bag Telecom.

It's very easy to see why customers do / don't get their advertised max line speeds, and FTTH certainly isn't the straight answer, as most of the congestion in NZ currently occurs between the exchange and the international circuits (Where users are on ADSL1), or in what the overall bandwidth pool that smaller ISP's have wholesaled from the larger ISPs.

An excellent graph, great explanation to go along with it, well done :) 


Comment by ojala, on 18-Nov-2010 05:35

@muppet: I didn't realize APE was so commercial, most IX'es around the world seem to be reasonably open with their statistics, membership details, and technical implementation.  In comparison, the finnish IX data is at http://stats.ficix.fi/total.html and peaks around 30 Gbit/s.  AMS-IX data is at http://www.ams-ix.net/statistics/ and peaks around 1.1 Tbit/s.  AMS-IX is where plenty of Europe goes through.

I agree that there is always a cost involved, even if "national traffic is free".  The question is more about getting the long-haul data pricing to a point where you'll get gigabits across the country just like renting a new room in the office.  In other words, just part of the overall expenses to operate the network.

Wrt to mandatory peering, the though was just about to get rid of the Telecom won't peer -problem.  I believe Telecom has an open peering policy outside of NZ which makes the situation pretty awkward to the other NZ ISP's.  When we founded FICIX, most IX's and ISP's had open peering policies and as international capacity was very expensive at the time, we wanted to make sure everyone peers at FICIX instead.  Incumbents did make other iSP's life difficult but not like that.  Nowadays it's the platform for peering and peering is not mandatory (but everyone does it).

@icepciknz: Ok, I never realized that the domestic capacity is so expensive and competition isn't happening countrywide.  The Wellington example just emphasize the issue, duopoly is much tougher.  Also explains why there are so many peering points in NZ.

I remember seeing a fiber map for NZ and I thought that the situation was much better.

Is there any new domestic long-haul fiber arriving?  Power companies?  Railroads?  Here power companies made a big difference in early 90's (the network is part of TDC now).

This also explains why the national traffic isn't happening -- and most likely won't happen until the bottleneck is gone, for both the iSP's and customers.


Comment by ojala, on 18-Nov-2010 05:57

Just for comparison I checked one source for current pricing and they use two carriers, one charges about 1400 NZD/month for GE metro access and the other charges 800 NZD/month for GE metro.  They use the latter within the region (comparable to GE within Akl or Wgtn) and the former is a countrywide.

You need one access port everywhere you want to connect, 2800 NZD/month for GE across the country, 1600 NZD/month for GE within the region.

There's no CIR but the carriers say that there is "enough capacity" and "they'll add more if necessary".  Makes sense with a metro ethernet service.


Comment by webnation, on 17-Dec-2010 13:47

this might be a obvious question to you, but i really want to understand the whole idea behind data caps. When anyone calculate it, it always assumes everyone will use the bandwidth 24/7 and calculate a price of /GB for that, which seems very irrational? let's get one thing straight, ISP dont pay for /GB right? they pay per bandwidth? I read somewhere isp put up a datacap for NZdler because it's a easy way to manage their bandwidth and also makes money if their margin is so limited as a reseller. How much international bandwidth purchased by ISP are impossible to get, otherwise consumers will surely know which ISP they should go to.


Add a comment

Please note: comments that are inappropriate or promotional in nature will be deleted. E-mail addresses are not displayed, but you must enter a valid e-mail address to confirm your comments.

Are you a registered Geekzone user? Login to have the fields below automatically filled in for you and to enable links in comments. If you have (or qualify to have) a Geekzone Blog then your comment will be automatically confirmed and shown in this blog post.

Your name:

Your e-mail:

Your webpage:

muppet's profile

TiM 
New Zealand


My name's Tim and I'm a Network Engineer! Well these days I'm actually a Network Architect. I'm one of those weird people that really love their job. I work with a great bunch of people in an interesting industry that's always changing. Sometimes slower than I'd like.

In my spare time I work on the LiCe Script for the EPIC5 IRC Client. Try it out, it's like Irssi but better.