Geekzone: technology news, blogs, forums
Guest
Welcome Guest.
You haven't logged in yet. If you don't have an account you can register now.


165 posts

Master Geek
+1 received by user: 18


Topic # 96667 1-Feb-2012 23:04
Send private message

I thought I would recount what I have just wasted 5 hours over the last 3 days on for amusement and perhaps to save someone else the same grief...

I am shifting web hosts. One of my sites I duplicated on the new server but wanted to test before pointing DNS to the new server so I added an entry to my hosts file. Unfortunately this did not work and I spent a lot of time running in circles putting test files on both servers to easily identify them, stuffing this up due to my ftp client following the hosts directive etc etc etc.

The problem I was having was that no matter what I did, visiting the site always resulted in the old (live) server responding. I even was able to see the new server's IP in a firefox add-on that displays the IP of the current site being visited, or with wget. I also could ping the url and get responses from the correct IP. Only my web browsers appeared to be misbehaving. However this was true on multiple computers running Windows 7, Mac Os Lion and Ubuntu 10.10. Eventually I found that if I tunneled my internet through my VPS or tethered my cellphone, the problem went away. A friend finally tipped me off to the possibility of a transparent proxy and a quick tweak to enable SSL, soon proved this to be true.

Some searches (with the benefit of hindsight) show that this proxy has been in place for some time (and there was a lot of talk about it in 2010) but I was not aware of it until now. Pity about the 5 hours but I guess I've learned something new today. Embarassed  Part of the reason that it never occurred to me is that I had figured a proxy would just cache whatever IP I visited, not override my requested IP by performing its own DNS query on the referrer!! Although I am (clearly) not very experienced with proxies, this really seems to me to be a very very bad way of implementing one. Damn you Telstraclear!!!   

Create new topic
27140 posts

Uber Geek
+1 received by user: 6579

Moderator
Trusted
Biddle Corp
Lifetime subscriber

  Reply # 576149 2-Feb-2012 06:17
Send private message

TCL have had a transparent proxy for ~10 years now and what you experienced is the standard way a transparent proxy works, you'll get the exact same issues on many other ISP's who also use transparent proxys.






2584 posts

Uber Geek
+1 received by user: 5

Mod Emeritus
Trusted
Lifetime subscriber

  Reply # 576152 2-Feb-2012 06:47
Send private message

sbiddle: TCL have had a transparent proxy for ~10 years now and what you experienced is the standard way a transparent proxy works, you'll get the exact same issues on many other ISP's who also use transparent proxys.


So just how many ISP's use transparent proxy's? I was looking at Unblock-us the other week which is a DNS based unblocking service and they only have a handful of ISP's listed using them globally.

http://support.unblock-us.com/customer/portal/articles/291561







Media centre PC - Case Silverstone LC16M with 2 X 80mm AcoustiFan DustPROOF, MOBO Gigabyte MA785GT-UD3H, CPU AMD X2 240 under volted, RAM 4 Gig DDR3 1033, HDD 120Gig System/512Gig data, Tuners 2 X Hauppauge HVR-3000, 1 X HVR-2200, Video Palit GT 220, Sound Realtek 886A HD (onboard), Optical LiteOn DH-401S Blue-ray using TotalMedia Theatre Power Corsair VX Series, 450W ATX PSU OS Windows 7 x64

677 posts

Ultimate Geek
+1 received by user: 27

Trusted

  Reply # 576182 2-Feb-2012 08:56
Send private message

Nety:
sbiddle: TCL have had a transparent proxy for ~10 years now and what you experienced is the standard way a transparent proxy works, you'll get the exact same issues on many other ISP's who also use transparent proxys.


So just how many ISP's use transparent proxy's? I was looking at Unblock-us the other week which is a DNS based unblocking service and they only have a handful of ISP's listed using them globally.

http://support.unblock-us.com/customer/portal/articles/291561


Transparent proxy cache is very common in ISP's. Not all are doing HTTP, some are specifically for P2P though, just depends on what the ISP sees as providing the most value.

Issues like the one seen here are very common also when you dont use the DNS servers of your ISP, you can have random broken internet issues with things like YouTube, Akamai and any other CDN as well as hosts file having no effect...

You can in many instances ask the ISP to not put your connection through the proxy, not sure if that is true for TCL though.






meat popsicle

129 posts

Master Geek
+1 received by user: 16

Trusted

  Reply # 576305 2-Feb-2012 13:47
Send private message

Another question, did the site/pages have cache control statements embedded to ensure the transparent proxy knows when to refresh the content or discard the cached copy?  HTTP meta tags etc.

I might be guilty of assuming those statements were absent, if so, I retract and withdraw... :)

a quick search turned up this useful (at initial glance) site.
[url=http://www.mnot.net/cache_docs/#CONTROL]





"Customers don’t expect you to be perfect. They do expect you to fix things when they go wrong." Donald Porter – British Airways

The views expressed here are my own and are not reflective of other organisms or organisations.

677 posts

Ultimate Geek
+1 received by user: 27

Trusted

  Reply # 576309 2-Feb-2012 13:53
Send private message

In the OP's case that will only stop stale content from the current prod server, he would have still been getting responses from the old server until DNS is changed to point to the new IP.





meat popsicle

8027 posts

Uber Geek
+1 received by user: 387

Trusted
Subscriber

  Reply # 576381 2-Feb-2012 17:23
Send private message

If the web host has a ssh server supports a SSH connection you can bypass the transparent proxy via ssh tunnel to the server and your local dns entry should work.

Just need a client like Putty for windows to send your web browser request over the ssh tunnel.

1466 posts

Uber Geek
+1 received by user: 147

Subscriber

  Reply # 576384 2-Feb-2012 17:39
Send private message

Nety:
sbiddle: TCL have had a transparent proxy for ~10 years now and what you experienced is the standard way a transparent proxy works, you'll get the exact same issues on many other ISP's who also use transparent proxys.


So just how many ISP's use transparent proxy's? I was looking at Unblock-us the other week which is a DNS based unblocking service and they only have a handful of ISP's listed using them globally.

http://support.unblock-us.com/customer/portal/articles/291561


I use unblock-us.com on Telecom, and it works 100% for Hulu and Netflix.

677 posts

Ultimate Geek
+1 received by user: 27

Trusted

  Reply # 576513 2-Feb-2012 21:15
Send private message

Ragnor: If the web host has a ssh server supports a SSH connection you can bypass the transparent proxy via ssh tunnel to the server and your local dns entry should work.

Just need a client like Putty for windows to send your web browser request over the ssh tunnel.


I always forget to mention this.
If you use the ssh option -D1040 or any port you like you can set the socks proxy configuration in your browser to use localhost and the port number after the -D then you dont need to browse to localhost to test the remote site.
With Putty you just specify a dynamic proxy port, it can be a simple way to test things.




meat popsicle



165 posts

Master Geek
+1 received by user: 18


  Reply # 576529 2-Feb-2012 21:29
Send private message

ptinson: In the OP's case that will only stop stale content from the current prod server, he would have still been getting responses from the old server until DNS is changed to point to the new IP. 



Correct. I wanted to do some testing before updating my DNS for the rest of the world. As for what the cache controls were, I haven't the foggiest as I've never used them.

 

RmACK: Eventually I found that if I tunneled my internet through my VPS or tethered my cellphone, the problem went away.

ragnor & ptinson, by tunneling through my VPS, I did indeed mean that I used ssh as a socks proxy

677 posts

Ultimate Geek
+1 received by user: 27

Trusted

  Reply # 576537 2-Feb-2012 21:37
Send private message

Ah i missed that bit, so i told you how to suck an egg really. Good for me.




meat popsicle



165 posts

Master Geek
+1 received by user: 18


  Reply # 576538 2-Feb-2012 21:39
Send private message

No worries :)

14216 posts

Uber Geek
+1 received by user: 2572

Trusted
Subscriber

  Reply # 576679 3-Feb-2012 09:48
Send private message

A VPN would get around this. I use Astrill.




AWS Certified Solution Architect Professional, Sysop Administrator Associate, and Developer Associate
TOGAF certified enterprise architect
Professional photographer


Create new topic

Twitter »

Follow us to receive Twitter updates when new discussions are posted in our forums:



Follow us to receive Twitter updates when news items and blogs are posted in our frontpage:



Follow us to receive Twitter updates when tech item prices are listed in our price comparison site:



Geekzone Live »

Try automatic live updates from Geekzone directly in your browser, without refreshing the page, with Geekzone Live now.



Are you subscribed to our RSS feed? You can download the latest headlines and summaries from our stories directly to your computer or smartphone by using a feed reader.

Alternatively, you can receive a daily email with Geekzone updates.