Geekzone: technology news, blogs, forums
Guest
Welcome Guest.
You haven't logged in yet. If you don't have an account you can register now.



137 posts

Master Geek
Inactive user


Topic # 106642 28-Jul-2012 12:09 Send private message

Hi,

Me again :)

Curious if ISP publish any information on how they manipulate your Internet traffic. So stuff like redirection, proxying, throttling, path manipulation and packet manipulation.

Any info appreciated.

View this topic in a long page with up to 500 replies per page Create new topic
 1 | 2
BDFL
49937 posts

Uber Geek
+1 received by user: 4624

Administrator
Trusted
Geekzone
Subscriber

  Reply # 663428 28-Jul-2012 21:08 Send private message

I have no knowledge of ISPs in New Zealand making this kind of information public.




1912 posts

Uber Geek
+1 received by user: 456

Trusted
Spark NZ

  Reply # 663445 28-Jul-2012 21:50 Send private message

Many ISPs do some of the things you list, primarily to improve performance to end users since content served from 100k away is going to be faster than content served from 2000k away. They are often complex and do require constant tuning but when they are working properly (most of the time, and most people never notice anything except better speeds) they are fantastic.

However, apart from some ISPs making very general statements about proxying, I don't believe that anyone provides details of what they do - partly for competitive reasons and partly because these things can change frequently in response to issues or changing traffic patterns and requirements.

The throttling that occurs with a small number of Telecom plans when a users goes over their monthly limit is an exception and it's clearly defined.

Short answer, most do at least some of the things, pretty much no-one will explain exactly what they are doing in public.

Cheers - N



137 posts

Master Geek
Inactive user


  Reply # 664229 30-Jul-2012 20:59 Send private message

Talkiet: Many ISPs do some of the things you list, primarily to improve performance to end users since content served from 100k away is going to be faster than content served from 2000k away. They are often complex and do require constant tuning but when they are working properly (most of the time, and most people never notice anything except better speeds) they are fantastic.

However, apart from some ISPs making very general statements about proxying, I don't believe that anyone provides details of what they do - partly for competitive reasons and partly because these things can change frequently in response to issues or changing traffic patterns and requirements.

The throttling that occurs with a small number of Telecom plans when a users goes over their monthly limit is an exception and it's clearly defined.

Short answer, most do at least some of the things, pretty much no-one will explain exactly what they are doing in public.

Cheers - N


Ok, thank you for your prompt and concise reply.  Is there any way you can have a say in what gets proxied, cached, redirected to local mirrors?  Or does any ISP offer something like this?  Or does any ISP NOT do any of this?

1332 posts

Uber Geek
+1 received by user: 152
Inactive user


  Reply # 664255 30-Jul-2012 21:26 Send private message

I know that Hosting Direct do not proxy/cache/filter any traffic on their network. I imagine this is true of many of the smaller ISPs as maintaining cache/proxy farms can be expensive.

2010 posts

Uber Geek
+1 received by user: 273

Trusted
Subscriber

  Reply # 664273 30-Jul-2012 21:53 Send private message

bradi:
Talkiet: Many ISPs do some of the things you list, primarily to improve performance to end users since content served from 100k away is going to be faster than content served from 2000k away. They are often complex and do require constant tuning but when they are working properly (most of the time, and most people never notice anything except better speeds) they are fantastic.

However, apart from some ISPs making very general statements about proxying, I don't believe that anyone provides details of what they do - partly for competitive reasons and partly because these things can change frequently in response to issues or changing traffic patterns and requirements.

The throttling that occurs with a small number of Telecom plans when a users goes over their monthly limit is an exception and it's clearly defined.

Short answer, most do at least some of the things, pretty much no-one will explain exactly what they are doing in public.

Cheers - N


Ok, thank you for your prompt and concise reply.  Is there any way you can have a say in what gets proxied, cached, redirected to local mirrors?  Or does any ISP offer something like this?  Or does any ISP NOT do any of this?


Typically not as for example the ISP has zero say in what Akamai or Google have in their caching nodes. It changes dynamically over time based on demand and on who's paying to have their content pushed out closer to the end user. Some ISPs don't even purchase their own bandwidth so their policy would be dictated by their host ISP.

From the ISPs point of view, the more content they can deliver locally the less $$ spent on international transit. However these days with the price of international bandwidth it's sometimes more cost effective to simply throw more bandwidth at the problem than invest in BlueCoat and PeerApp farms.

For small to medium ISPs it can be tough to get the hit ratios out of caches as they may not have enough users seeding them, so for some it's pointless. Exetel in Australia posted some interesting blog entries when they looked into this once they grew to the critical size. Read about it here

I'll add one last comment in here, ISPs which say they do nothing to their users traffic typically haven't got the traffic levels to justify it, or haven't yet invested it the tech. I remember how Xnet used to harp on about how traffic was never manipulated... try ask them now, I'm sure they wouldn't be able to provide a usable service without some packet wrangling. 



137 posts

Master Geek
Inactive user


  Reply # 664277 30-Jul-2012 22:07 Send private message

insane:
bradi:
Talkiet: Many ISPs do some of the things you list, primarily to improve performance to end users since content served from 100k away is going to be faster than content served from 2000k away. They are often complex and do require constant tuning but when they are working properly (most of the time, and most people never notice anything except better speeds) they are fantastic.

However, apart from some ISPs making very general statements about proxying, I don't believe that anyone provides details of what they do - partly for competitive reasons and partly because these things can change frequently in response to issues or changing traffic patterns and requirements.

The throttling that occurs with a small number of Telecom plans when a users goes over their monthly limit is an exception and it's clearly defined.

Short answer, most do at least some of the things, pretty much no-one will explain exactly what they are doing in public.

Cheers - N


Ok, thank you for your prompt and concise reply.  Is there any way you can have a say in what gets proxied, cached, redirected to local mirrors?  Or does any ISP offer something like this?  Or does any ISP NOT do any of this?


Typically not as for example the ISP has zero say in what Akamai or Google have in their caching nodes. It changes dynamically over time based on demand and on who's paying to have their content pushed out closer to the end user. Some ISPs don't even purchase their own bandwidth so their policy would be dictated by their host ISP.

From the ISPs point of view, the more content they can deliver locally the less $$ spent on international transit. However these days with the price of international bandwidth it's sometimes more cost effective to simply throw more bandwidth at the problem than invest in BlueCoat and PeerApp farms.

For small to medium ISPs it can be tough to get the hit ratios out of caches as they may not have enough users seeding them, so for some it's pointless. Exetel in Australia posted some interesting blog entries when they looked into this once they grew to the critical size. Read about it here

I'll add one last comment in here, ISPs which say they do nothing to their users traffic typically haven't got the traffic levels to justify it, or haven't yet invested it the tech. I remember how Xnet used to harp on about how traffic was never manipulated... try ask them now, I'm sure they wouldn't be able to provide a usable service without some packet wrangling. 


Awesome, thanks heaps mate.  Understood, orders of scale and all that.  For me this makes very little difference, I don't care about stuff the way most people do, don't really use Facebook, YouTube, etc and I suspect my traffic profile would be seriously out of the norm...

Does this mean the move to UFB is likely to increase caches rather than international transit?  Does this mean that smaller ISPs will have no choice but to buy capacity and policy from tier one providers?

2010 posts

Uber Geek
+1 received by user: 273

Trusted
Subscriber

  Reply # 664296 30-Jul-2012 22:36 Send private message

bradi: 

Awesome, thanks heaps mate.  Understood, orders of scale and all that.  For me this makes very little difference, I don't care about stuff the way most people do, don't really use Facebook, YouTube, etc and I suspect my traffic profile would be seriously out of the norm...

Does this mean the move to UFB is likely to increase caches rather than international transit?  Does this mean that smaller ISPs will have no choice but to buy capacity and policy from tier one providers?


I think it's fair to say that caching or hosting of content locally will be a must as you'd be surprised how average a high speed connection performs when content is pulled from across the other side of the world.

Smaller ISPs already pretty much have to purchase from NZs Tier 1 ISPs as they can't afford to pony up the $$ needed to purchase bandwidth directly from SX. Telstraclear for example doesn't force it's caching policy on their transit customers though, but maybe they could provide that if someone actually wanted that.



137 posts

Master Geek
Inactive user


  Reply # 664297 30-Jul-2012 22:37 Send private message

Talkiet: Many ISPs do some of the things you list, primarily to improve performance to end users since content served from 100k away is going to be faster than content served from 2000k away. They are often complex and do require constant tuning but when they are working properly (most of the time, and most people never notice anything except better speeds) they are fantastic.

However, apart from some ISPs making very general statements about proxying, I don't believe that anyone provides details of what they do - partly for competitive reasons and partly because these things can change frequently in response to issues or changing traffic patterns and requirements.

The throttling that occurs with a small number of Telecom plans when a users goes over their monthly limit is an exception and it's clearly defined.

Short answer, most do at least some of the things, pretty much no-one will explain exactly what they are doing in public.

Cheers - N


I'm kinda inclined to disagree... I have a host in Europe that I can download from faster (over a VPN) than I can get content from my ISPs mirror.  So caching is moot when you have bandwidth.  Proxying, caching and general manipulation of traffic only really benefits the ISP.

1912 posts

Uber Geek
+1 received by user: 456

Trusted
Spark NZ

  Reply # 664317 30-Jul-2012 23:14 Send private message

bradi: I'm kinda inclined to disagree... I have a host in Europe that I can download from faster (over a VPN) than I can get content from my ISPs mirror.  So caching is moot when you have bandwidth.  Proxying, caching and general manipulation of traffic only really benefits the ISP.


You might want to disagree... But the way TCP works isn't hugely dependent on your belief. You can fudge things with many concurrent threads, but nearby traffic sources always have the capability to go faster than faraway ones. What well setup proxying does is serve stuff from closer to you whenever possible.

If you want to compare apples to oranges though then yes I agree - it will be possible to get a multithreaded download of a linux iso (HAH!) from your single user seedbox in Germany at a higher aggregate throughput than from a single threaded download from a server in Wellington that might be serving a large number of customers simultaneously.

That doesn't prove that caching isn't valuable, it proves that by controlling both ends of a connection you can normally optimse things really well.

Cheers - N


2509 posts

Uber Geek
+1 received by user: 243

Trusted
Subscriber

  Reply # 664347 31-Jul-2012 01:24 Send private message

bradi: I'm kinda inclined to disagree... I have a host in Europe that I can download from faster (over a VPN) than I can get content from my ISPs mirror.  So caching is moot when you have bandwidth.  Proxying, caching and general manipulation of traffic only really benefits the ISP.


You realise that you're arguing with no less than two employees of major telcos who live and breathe this stuff right?  You may observe different behaviour when you manipulate the factors deciding the speed of your connection, but they're simply telling it to you like it is.

And for what it's worth, in order to disprove your anecdote; I have a host in the US and my server is on a 100Mb/s dedicated upstream port, and locally cached content from my ISP is faster than downloading from my own web server.

You could try and argue that my ISP is "manipulating" my connection to make it seem faster, but Orcon has in the past been rather upfront about their traffic management practices, and one specific point was that business connections are not managed, and have priority QoS.  Mine is a business connection, incidentally.



137 posts

Master Geek
Inactive user


  Reply # 664952 31-Jul-2012 21:16 Send private message

Talkiet:
bradi: I'm kinda inclined to disagree... I have a host in Europe that I can download from faster (over a VPN) than I can get content from my ISPs mirror. So caching is moot when you have bandwidth. Proxying, caching and general manipulation of traffic only really benefits the ISP.


You might want to disagree... But the way TCP works isn't hugely dependent on your belief. You can fudge things with many concurrent threads, but nearby traffic sources always have the capability to go faster than faraway ones. What well setup proxying does is serve stuff from closer to you whenever possible.

If you want to compare apples to oranges though then yes I agree - it will be possible to get a multithreaded download of a linux iso (HAH!) from your single user seedbox in Germany at a higher aggregate throughput than from a single threaded download from a server in Wellington that might be serving a large number of customers simultaneously.

That doesn't prove that caching isn't valuable, it proves that by controlling both ends of a connection you can normally optimse things really well.

Cheers - N



Hey, definitely not disputing the fundamentals of IP (which is much less "fudged" than TCP) and is basically what transports packets around the Internet. And yes totally agree that IF the content is local then the benefits of less hops to your destination SHOULD in theory hold true.

Where this falls over (and where I disagree) is when

a). the content you want can't be proxied... with the exception of HTTP most content can't be proxied easily... and the content you want isn't local (surprisingly over 99% of the Internet is NOT based in NZ (shock!)... or in your particular ISPs cache (horror!))... and it changes every minute

b). Caching is put forward as the "only" solution to "poor old NZ being at the bottom of the world away from the majority of content"... curious how Japan manages this... or Hawaii, feel free to comment :)

c). Invasive technology is implemented in order to satisfy a perceived "bandwidth" problem... feel free to comment on how much bandwidth is available now, and in the near future into NZ and how ISPs rate limit their customers even when accessing on net or local content if you have any info here

d). Governments put public money in to improving "local" bandwidth without addressing the "next hop" problem (which really only benefits the big content providers)

BTW can we leave the guessing of what I use my bandwidth for to my ISP... I could care less what you (or anyone else) use your bandwidth for, and it's none of your business (but I think you'd be surprised ;P).



137 posts

Master Geek
Inactive user


  Reply # 664954 31-Jul-2012 21:20 Send private message

Kyanar:
bradi: I'm kinda inclined to disagree... I have a host in Europe that I can download from faster (over a VPN) than I can get content from my ISPs mirror.? So caching is moot when you have bandwidth.? Proxying, caching and general manipulation of traffic only really benefits the ISP.


You realise that you're arguing with no less than two employees of major telcos who live and breathe this stuff right?


Yea, me too ;)



137 posts

Master Geek
Inactive user


  Reply # 664958 31-Jul-2012 21:27 Send private message

Kyanar: And for what it's worth, in order to disprove your anecdote; I have a host in the US and my server is on a 100Mb/s dedicated upstream port, and locally cached content from my ISP is faster than downloading from my own web server.

You could try and argue that my ISP is "manipulating" my connection to make it seem faster, but Orcon has in the past been rather upfront about their traffic management practices, and one specific point was that business connections are not managed, and have priority QoS.? Mine is a business connection, incidentally.


Sorry mate couldn't resist this one....

Doesn't this just prove that caches are bollocks :P

You can't cache the Internet

1912 posts

Uber Geek
+1 received by user: 456

Trusted
Spark NZ

  Reply # 665027 31-Jul-2012 22:41 Send private message

bradi:
Hey, definitely not disputing the fundamentals of IP (which is much less "fudged" than TCP) and is basically what transports packets around the Internet. And yes totally agree that IF the content is local then the benefits of less hops to your destination SHOULD in theory hold true.


Latency, not hops.

Where this falls over (and where I disagree) is when

a). the content you want can't be proxied... with the exception of HTTP most content can't be proxied easily... and the content you want isn't local (surprisingly over 99% of the Internet is NOT based in NZ (shock!)... or in your particular ISPs cache (horror!))... and it changes every minute


By overall volume, most content is proxiable, or can be delivered through CDNs. This is a fact, not an opinion. Note I say proxiable, not proxied-  I'm not saying anything about the current state of proxying at any retail ISP...

b). Caching is put forward as the "only" solution to "poor old NZ being at the bottom of the world away from the majority of content"... curious how Japan manages this... or Hawaii, feel free to comment :)


Actually, Japan is a great example. In fact any country that has their own strongly entrenched culture (and more specifically language) that isn't shared widely around the globe has a stronger ratio of local to internationally sourced content. Hawaii is about 4100k from LA, Auckland is about 10500. Still Caching is NOT put forward as the only solution. CDNs are _wonderful_ things and they work brilliantly.

c). Invasive technology is implemented in order to satisfy a perceived "bandwidth" problem... feel free to comment on how much bandwidth is available now, and in the near future into NZ and how ISPs rate limit their customers even when accessing on net or local content if you have any info here


Heh, Of course I'm not going to comment on a commercially sensitive topic as current and future bandwidth capacity, EVEN if I knew the up to the minute figures. What I will say is that my plain vanilla (nothing different from any other Telecom Retail customer) connection at home achieves the expected speeds 24*7, within the constraints of what TCP does with the latency and remote server capacity.

d). Governments put public money in to improving "local" bandwidth without addressing the "next hop" problem (which really only benefits the big content providers)


The government money is going to benefit everyone, not just through UFB but through RBI as well. The "next hop" problem can't be solved unless you can figure out how to make light go faster than it does already, or some cunning way to drill straight through the earth rather than around it :-) ... CDNs and Proxies don't just need to benefit the big content providers either. For example, my personal website (www.nzsnaps.com) is on a US server and for most people in NZ is served off a CDN substantially closer than that.

BTW can we leave the guessing of what I use my bandwidth for to my ISP... I could care less what you (or anyone else) use your bandwidth for, and it's none of your business (but I think you'd be surprised ;P).


Completely fair comment - but if I had a buck for every time someone told me they needed to download Linux ISOs every day, I'd have more money than Bill Gates - mostly cos he'd be bankrupt cos everyone was using Linux. It wasn't a dig at you at all - it was my generic tongue in cheek term for any large download.

Cheers - N


wjw

158 posts

Master Geek


  Reply # 665927 2-Aug-2012 10:15 Send private message

As Neil mentioned before the biggest issue we have in NZ is the latency between us and the content sources. This article is a good starting point in understanding that for those that don't.

http://www.igvita.com/2012/07/19/latency-the-new-web-performance-bottleneck/

The only way round this issue is to use multiple TCP sessions. 

I fight with this every day and it's an uphill battle to explain why their Sharepoint (based in Denver) is slow.

 1 | 2
View this topic in a long page with up to 500 replies per page Create new topic




Twitter »
Follow us to receive Twitter updates when new discussions are posted in our forums:



Follow us to receive Twitter updates when news items and blogs are posted in our frontpage:



Follow us to receive Twitter updates when tech item prices are listed in our price comparison site:





Trending now »

Hot discussions in our forums right now:

Speed limit when overtaking? Teach me please.
Created by nakedmolerat, last reply by joker97 on 25-Oct-2014 09:00 (75 replies)
Pages... 3 4 5


House Auctions
Created by t0ny, last reply by mattwnz on 25-Oct-2014 00:18 (36 replies)
Pages... 2 3


VDSL, which router/modem sub $200?
Created by TeaLeaf, last reply by TeaLeaf on 24-Oct-2014 23:26 (16 replies)
Pages... 2


30 too old to get into IT?
Created by Interslice, last reply by shk292 on 24-Oct-2014 20:39 (16 replies)
Pages... 2


American legal jurisdiction in New Zealand
Created by ajobbins, last reply by gzt on 21-Oct-2014 14:58 (30 replies)
Pages... 2


Neon - Sky's new streaming service
Created by JarrodM, last reply by kharris on 25-Oct-2014 09:50 (24 replies)
Pages... 2


iPad Air 2 and iPad Mini 3. Gonna get one?
Created by Dingbatt, last reply by tdgeek on 25-Oct-2014 01:10 (109 replies)
Pages... 6 7 8


5Ghz AP recommendations?
Created by ubergeeknz, last reply by sbiddle on 24-Oct-2014 12:42 (12 replies)


Geekzone Live »

Try automatic live updates from Geekzone directly in your browser, without refreshing the page, with Geekzone Live now.

Are you subscribed to our RSS feed? You can download the latest headlines and summaries from our stories directly to your computer or smartphone by using a feed reader.

Alternatively, you can receive a daily email with Geekzone updates.