[TelstraClear] Mysterious hosts file ignoring turned out to be a transparent proxy...

Forums › One New Zealand (including Vodafone, WxC, Farmside) › [TelstraClear] Mysterious hosts file ignoring turned out to be a transparent proxy...

RmACK

196 posts

Master Geek

#96667 1-Feb-2012 23:04

I thought I would recount what I have just wasted 5 hours over the last 3 days on for amusement and perhaps to save someone else the same grief...

I am shifting web hosts. One of my sites I duplicated on the new server but wanted to test before pointing DNS to the new server so I added an entry to my hosts file. Unfortunately this did not work and I spent a lot of time running in circles putting test files on both servers to easily identify them, stuffing this up due to my ftp client following the hosts directive etc etc etc.

The problem I was having was that no matter what I did, visiting the site always resulted in the old (live) server responding. I even was able to see the new server's IP in a firefox add-on that displays the IP of the current site being visited, or with wget. I also could ping the url and get responses from the correct IP. Only my web browsers appeared to be misbehaving. However this was true on multiple computers running Windows 7, Mac Os Lion and Ubuntu 10.10. Eventually I found that if I tunneled my internet through my VPS or tethered my cellphone, the problem went away. A friend finally tipped me off to the possibility of a transparent proxy and a quick tweak to enable SSL, soon proved this to be true.

Some searches (with the benefit of hindsight) show that this proxy has been in place for some time (and there was a lot of talk about it in 2010) but I was not aware of it until now. Pity about the 5 hours but I guess I've learned something new today.

Part of the reason that it never occurred to me is that I had figured a proxy would just cache whatever IP I visited, not override my requested IP by performing its own DNS query on the referrer!! Although I am (clearly) not very experienced with proxies, this really seems to me to be a very very bad way of implementing one. Damn you Telstraclear!!!

sbiddle

30853 posts

Uber Geek

Retired Mod

Trusted

Biddle Corp

Lifetime subscriber

#576149 2-Feb-2012 06:17

TCL have had a transparent proxy for ~10 years now and what you experienced is the standard way a transparent proxy works, you'll get the exact same issues on many other ISP's who also use transparent proxys.

Nety

2584 posts

Uber Geek

Retired Mod

Trusted

Lifetime subscriber

#576152 2-Feb-2012 06:47

sbiddle: TCL have had a transparent proxy for ~10 years now and what you experienced is the standard way a transparent proxy works, you'll get the exact same issues on many other ISP's who also use transparent proxys.

So just how many ISP's use transparent proxy's? I was looking at Unblock-us the other week which is a DNS based unblocking service and they only have a handful of ISP's listed using them globally.

http://support.unblock-us.com/customer/portal/articles/291561

Media centre PC - Case Silverstone LC16M with 2 X 80mm AcoustiFan DustPROOF, MOBO Gigabyte MA785GT-UD3H, CPU AMD X2 240 under volted, RAM 4 Gig DDR3 1033, HDD 120Gig System/512Gig data, Tuners 2 X Hauppauge HVR-3000, 1 X HVR-2200, Video Palit GT 220, Sound Realtek 886A HD (onboard), Optical LiteOn DH-401S Blue-ray using TotalMedia Theatre Power Corsair VX Series, 450W ATX PSU OS Windows 7 x64

ptinson

677 posts

Ultimate Geek

Trusted

#576182 2-Feb-2012 08:56

Nety:
sbiddle: TCL have had a transparent proxy for ~10 years now and what you experienced is the standard way a transparent proxy works, you'll get the exact same issues on many other ISP's who also use transparent proxys.

So just how many ISP's use transparent proxy's? I was looking at Unblock-us the other week which is a DNS based unblocking service and they only have a handful of ISP's listed using them globally.

http://support.unblock-us.com/customer/portal/articles/291561

Transparent proxy cache is very common in ISP's. Not all are doing HTTP, some are specifically for P2P though, just depends on what the ISP sees as providing the most value.

Issues like the one seen here are very common also when you dont use the DNS servers of your ISP, you can have random broken internet issues with things like YouTube, Akamai and any other CDN as well as hosts file having no effect...

You can in many instances ask the ISP to not put your connection through the proxy, not sure if that is true for TCL though.

meat popsicle

DoomlordVekk

129 posts

Master Geek

Trusted

#576305 2-Feb-2012 13:47

Another question, did the site/pages have cache control statements embedded to ensure the transparent proxy knows when to refresh the content or discard the cached copy? HTTP meta tags etc.

I might be guilty of assuming those statements were absent, if so, I retract and withdraw... :)

a quick search turned up this useful (at initial glance) site.
[url=http://www.mnot.net/cache_docs/#CONTROL]

"Customers don’t expect you to be perfect. They do expect you to fix things when they go wrong." Donald Porter – British Airways

The views expressed here are my own and are not reflective of other organisms or organisations.

ptinson

677 posts

Ultimate Geek

Trusted

#576309 2-Feb-2012 13:53

In the OP's case that will only stop stale content from the current prod server, he would have still been getting responses from the old server until DNS is changed to point to the new IP.

meat popsicle

Ragnor

8223 posts

Uber Geek

Trusted

#576381 2-Feb-2012 17:23

If the web host has a ssh server supports a SSH connection you can bypass the transparent proxy via ssh tunnel to the server and your local dns entry should work.

Just need a client like Putty for windows to send your web browser request over the ssh tunnel.

timbosan

2160 posts

Uber Geek

#576384 2-Feb-2012 17:39

Nety:
sbiddle: TCL have had a transparent proxy for ~10 years now and what you experienced is the standard way a transparent proxy works, you'll get the exact same issues on many other ISP's who also use transparent proxys.

So just how many ISP's use transparent proxy's? I was looking at Unblock-us the other week which is a DNS based unblocking service and they only have a handful of ISP's listed using them globally.

http://support.unblock-us.com/customer/portal/articles/291561

I use unblock-us.com on Telecom, and it works 100% for Hulu and Netflix.

Sharesies

Trade NZ and US shares and funds with Sharesies (affiliate link).

ptinson

677 posts

Ultimate Geek

Trusted

#576513 2-Feb-2012 21:15

Ragnor: If the web host has a ssh server supports a SSH connection you can bypass the transparent proxy via ssh tunnel to the server and your local dns entry should work.

Just need a client like Putty for windows to send your web browser request over the ssh tunnel.

I always forget to mention this.
If you use the ssh option -D1040 or any port you like you can set the socks proxy configuration in your browser to use localhost and the port number after the -D then you dont need to browse to localhost to test the remote site.
With Putty you just specify a dynamic proxy port, it can be a simple way to test things.

meat popsicle

RmACK

196 posts

Master Geek

#576529 2-Feb-2012 21:29

ptinson: In the OP's case that will only stop stale content from the current prod server, he would have still been getting responses from the old server until DNS is changed to point to the new IP.

Correct. I wanted to do some testing before updating my DNS for the rest of the world. As for what the cache controls were, I haven't the foggiest as I've never used them.