Hi,
Me again :)
Curious if ISP publish any information on how they manipulate your Internet traffic. So stuff like redirection, proxying, throttling, path manipulation and packet manipulation.
Any info appreciated.
![]() ![]() ![]() |
|
Please support Geekzone by subscribing, or using one of our referral links: Samsung | AliExpress | Wise | Sharesies | Hatch | GoodSync | Backblaze backup
Please note all comments are from my own brain and don't necessarily represent the position or opinions of my employer, previous employers, colleagues, friends or pets.
Talkiet: Many ISPs do some of the things you list, primarily to improve performance to end users since content served from 100k away is going to be faster than content served from 2000k away. They are often complex and do require constant tuning but when they are working properly (most of the time, and most people never notice anything except better speeds) they are fantastic.
However, apart from some ISPs making very general statements about proxying, I don't believe that anyone provides details of what they do - partly for competitive reasons and partly because these things can change frequently in response to issues or changing traffic patterns and requirements.
The throttling that occurs with a small number of Telecom plans when a users goes over their monthly limit is an exception and it's clearly defined.
Short answer, most do at least some of the things, pretty much no-one will explain exactly what they are doing in public.
Cheers - N
bradi:Talkiet: Many ISPs do some of the things you list, primarily to improve performance to end users since content served from 100k away is going to be faster than content served from 2000k away. They are often complex and do require constant tuning but when they are working properly (most of the time, and most people never notice anything except better speeds) they are fantastic.
However, apart from some ISPs making very general statements about proxying, I don't believe that anyone provides details of what they do - partly for competitive reasons and partly because these things can change frequently in response to issues or changing traffic patterns and requirements.
The throttling that occurs with a small number of Telecom plans when a users goes over their monthly limit is an exception and it's clearly defined.
Short answer, most do at least some of the things, pretty much no-one will explain exactly what they are doing in public.
Cheers - N
Ok, thank you for your prompt and concise reply. Is there any way you can have a say in what gets proxied, cached, redirected to local mirrors? Or does any ISP offer something like this? Or does any ISP NOT do any of this?
insane:bradi:Talkiet: Many ISPs do some of the things you list, primarily to improve performance to end users since content served from 100k away is going to be faster than content served from 2000k away. They are often complex and do require constant tuning but when they are working properly (most of the time, and most people never notice anything except better speeds) they are fantastic.
However, apart from some ISPs making very general statements about proxying, I don't believe that anyone provides details of what they do - partly for competitive reasons and partly because these things can change frequently in response to issues or changing traffic patterns and requirements.
The throttling that occurs with a small number of Telecom plans when a users goes over their monthly limit is an exception and it's clearly defined.
Short answer, most do at least some of the things, pretty much no-one will explain exactly what they are doing in public.
Cheers - N
Ok, thank you for your prompt and concise reply. Is there any way you can have a say in what gets proxied, cached, redirected to local mirrors? Or does any ISP offer something like this? Or does any ISP NOT do any of this?
Typically not as for example the ISP has zero say in what Akamai or Google have in their caching nodes. It changes dynamically over time based on demand and on who's paying to have their content pushed out closer to the end user. Some ISPs don't even purchase their own bandwidth so their policy would be dictated by their host ISP.
From the ISPs point of view, the more content they can deliver locally the less $$ spent on international transit. However these days with the price of international bandwidth it's sometimes more cost effective to simply throw more bandwidth at the problem than invest in BlueCoat and PeerApp farms.
For small to medium ISPs it can be tough to get the hit ratios out of caches as they may not have enough users seeding them, so for some it's pointless. Exetel in Australia posted some interesting blog entries when they looked into this once they grew to the critical size. Read about it here
I'll add one last comment in here, ISPs which say they do nothing to their users traffic typically haven't got the traffic levels to justify it, or haven't yet invested it the tech. I remember how Xnet used to harp on about how traffic was never manipulated... try ask them now, I'm sure they wouldn't be able to provide a usable service without some packet wrangling.
bradi:
Awesome, thanks heaps mate. Understood, orders of scale and all that. For me this makes very little difference, I don't care about stuff the way most people do, don't really use Facebook, YouTube, etc and I suspect my traffic profile would be seriously out of the norm...
Does this mean the move to UFB is likely to increase caches rather than international transit? Does this mean that smaller ISPs will have no choice but to buy capacity and policy from tier one providers?
Talkiet: Many ISPs do some of the things you list, primarily to improve performance to end users since content served from 100k away is going to be faster than content served from 2000k away. They are often complex and do require constant tuning but when they are working properly (most of the time, and most people never notice anything except better speeds) they are fantastic.
However, apart from some ISPs making very general statements about proxying, I don't believe that anyone provides details of what they do - partly for competitive reasons and partly because these things can change frequently in response to issues or changing traffic patterns and requirements.
The throttling that occurs with a small number of Telecom plans when a users goes over their monthly limit is an exception and it's clearly defined.
Short answer, most do at least some of the things, pretty much no-one will explain exactly what they are doing in public.
Cheers - N
bradi: I'm kinda inclined to disagree... I have a host in Europe that I can download from faster (over a VPN) than I can get content from my ISPs mirror. So caching is moot when you have bandwidth. Proxying, caching and general manipulation of traffic only really benefits the ISP.
Please note all comments are from my own brain and don't necessarily represent the position or opinions of my employer, previous employers, colleagues, friends or pets.
bradi: I'm kinda inclined to disagree... I have a host in Europe that I can download from faster (over a VPN) than I can get content from my ISPs mirror. So caching is moot when you have bandwidth. Proxying, caching and general manipulation of traffic only really benefits the ISP.
Talkiet:bradi: I'm kinda inclined to disagree... I have a host in Europe that I can download from faster (over a VPN) than I can get content from my ISPs mirror. So caching is moot when you have bandwidth. Proxying, caching and general manipulation of traffic only really benefits the ISP.
You might want to disagree... But the way TCP works isn't hugely dependent on your belief. You can fudge things with many concurrent threads, but nearby traffic sources always have the capability to go faster than faraway ones. What well setup proxying does is serve stuff from closer to you whenever possible.
If you want to compare apples to oranges though then yes I agree - it will be possible to get a multithreaded download of a linux iso (HAH!) from your single user seedbox in Germany at a higher aggregate throughput than from a single threaded download from a server in Wellington that might be serving a large number of customers simultaneously.
That doesn't prove that caching isn't valuable, it proves that by controlling both ends of a connection you can normally optimse things really well.
Cheers - N
Kyanar:bradi: I'm kinda inclined to disagree... I have a host in Europe that I can download from faster (over a VPN) than I can get content from my ISPs mirror.? So caching is moot when you have bandwidth.? Proxying, caching and general manipulation of traffic only really benefits the ISP.
You realise that you're arguing with no less than two employees of major telcos who live and breathe this stuff right?
Kyanar: And for what it's worth, in order to disprove your anecdote; I have a host in the US and my server is on a 100Mb/s dedicated upstream port, and locally cached content from my ISP is faster than downloading from my own web server.
You could try and argue that my ISP is "manipulating" my connection to make it seem faster, but Orcon has in the past been rather upfront about their traffic management practices, and one specific point was that business connections are not managed, and have priority QoS.? Mine is a business connection, incidentally.
bradi:
Hey, definitely not disputing the fundamentals of IP (which is much less "fudged" than TCP) and is basically what transports packets around the Internet. And yes totally agree that IF the content is local then the benefits of less hops to your destination SHOULD in theory hold true.
Where this falls over (and where I disagree) is when
a). the content you want can't be proxied... with the exception of HTTP most content can't be proxied easily... and the content you want isn't local (surprisingly over 99% of the Internet is NOT based in NZ (shock!)... or in your particular ISPs cache (horror!))... and it changes every minute
b). Caching is put forward as the "only" solution to "poor old NZ being at the bottom of the world away from the majority of content"... curious how Japan manages this... or Hawaii, feel free to comment :)
c). Invasive technology is implemented in order to satisfy a perceived "bandwidth" problem... feel free to comment on how much bandwidth is available now, and in the near future into NZ and how ISPs rate limit their customers even when accessing on net or local content if you have any info here
d). Governments put public money in to improving "local" bandwidth without addressing the "next hop" problem (which really only benefits the big content providers)
BTW can we leave the guessing of what I use my bandwidth for to my ISP... I could care less what you (or anyone else) use your bandwidth for, and it's none of your business (but I think you'd be surprised ;P).
Please note all comments are from my own brain and don't necessarily represent the position or opinions of my employer, previous employers, colleagues, friends or pets.
|
![]() ![]() ![]() |