[TelstraClear] TelstraClear transparent proxy, OpenDNS

Forums › One New Zealand (including Vodafone, WxC, Farmside) › [TelstraClear] TelstraClear transparent proxy, OpenDNS

1 | 2 | 3

30853 posts

Uber Geek

Retired Mod

Trusted

Biddle Corp

Lifetime subscriber

#192385 27-Jan-2009 21:21

I find theage.com.au regularly has all sorts of issues when accessing it via TCL. Sometimes the whole page won't load and plenty of times I get the same timeouts on some of the ads. It's a real pain.

freitasm

BDFL - Memuneh
79288 posts

Uber Geek

Administrator

ID Verified

Trusted

Geekzone

Lifetime subscriber

#193749 3-Feb-2009 17:59

Olof has posted a comment update in my blog.

freitasm

BDFL - Memuneh
79288 posts

Uber Geek

Administrator

ID Verified

Trusted

Geekzone

Lifetime subscriber

#194122 4-Feb-2009 22:05

Folks, we've got an update - and I will post a new blog entry later.

As stated a patch is being released Thursday to address some proxy problems, and it could be the fix for the TCP errors we've seen.

As for OpenDNS, we have identified that this problem happens when TelstraClear users try to access content from Akamai servers.

There's a discussion on OpenDNS forums explaining how when a DNS request is made Akamai will return an IP that's closer to the user. When querying OpenDNS the IP will be the one closer to the node.

The client browser will request the content using the IP provided by openDNS and it seems that at certain point Akamai decides to send the packets back through the nearest node to the destination.

At this point a stateful firewall on their border will drop the packet since the request didn't come through that firewall.

If this is the case, the easiest fix would be for OpenDNS to have some nodes down here...

I will post this in my blog later. Big thanks to TelstraClear and Olof who worked on this even through the late hours...

dub

2 posts

Wannabe Geek
Inactive user

#195286 11-Feb-2009 11:23

If this is the case, the easiest fix would be for OpenDNS to have some nodes down here...

Unless they have a node at every ISP that hosts an Akamai deployment I expect the "problem" would remain, in some form.

Has your experience changed since the work they did on the caches last Thursday.

olof

154 posts

Master Geek

Trusted

One NZ

#195721 13-Feb-2009 17:46

All,

We have been able to isolate the issue to occur in the following circumstances as it relates to TelstraClear customers in New Zealand:

1. Customer is using a TelstraClear access method that is transparently cached
2. Customer has the OpenDNS US based DNS servers configured
3. Customer attempts to access site that uses geographic load-balancing (Akamized sites for example)

Under the above circumstances, the customer is unlikely to be able to access the web site.

The explanation and root cause of the issue is as follows. Let's use a fictional web site "shoerax.com", that uses Akamai to serve its content:

1) Customer's PC looks up "shoerax.com" against the OpenDNS DNS servers based in the US

2) shoerax.com uses Akamai, and Akamai DNS servers. Hence the customer's DNS lookup goes to the OpenDNS DNS servers in the US, and these servers recursively look up "shoerax.com" from the Akamai DNS servers. As Akamai uses geographic load-balancing they attempt to return the IPs of servers that "are close to the customer". Akamai use the source IP of the querying recursive DNS server (= OpenDNS US based servers in this instance) to determine what "close to the customer is". Normally this works fine, as most people use recursive DNS servers that are close to where they are. However, when NZ customers use US based DNS servers (like OpenDNS), things break down a little, as US based server IPs are returned. (And they are obviously not close to the customer!) Hence, the Akamai DNS returns IP addresses of US based Akamai servers for shoerax.com.

3) The customer's PC attempts to establish an HTTP connection with the US based Akamai servers. (Using the IP addresses returned by OpenDNS.)

4) The TelstraClear transparent caches intercept the HTTP traffic. The transparent caches look up shoerax.com against TelstraClear DNS servers. As these DNS servers are in NZ, Akamai correctly returns the IP addresses of NZ based Akamai servers.

5) The caches attempt to connect to the NZ Akamai servers to retrieve the web objects.

6) The NZ Akamai servers sends return packets to the customers PC's (as this is the source IP of the incoming traffic). This is where the problem is. The return traffic should have gone to the caches and not to the customer's PC.

7) Caches time out

8) Customer's browser times out

The TelstraClear caches are situated on the NZ side of our international circuits and only attempt to cache international traffic. They expect that egress international HTTP traffic has return traffic coming back on the international circuits.

The issue with OpenDNS and TelstraClear transparent caching in conjunction with geographically load-balanced sites, is that the traffic is initially sent internationally (due to lookups against US based OpenDNS servers) and then the caches sends the traffic back domestically (due to lookups against NZ based TelstraClear servers). Due to this asymmetry, return traffic from the target web site do not reach the caches.

The above is unfortunate, as OpenDNS is an _excellent_ service. Until this issue has been resolved, we recommend that TelstraClear customers do not use overseas DNS servers, like OpenDNS, as this will cause issues with geographically load-balanced sites.

I have offered to workwith OpenDNS to come up with a solution.

--Olof

Ps. For optimum performance, we recommend that TelstraClear customers use the following DNS servers: 203.97.78.43 and 203.97.78.44.

Screeb

698 posts

Ultimate Geek

#195735 13-Feb-2009 19:04

olof:
Ps. For optimum performance, we recommend that TelstraClear customers use the following DNS servers: 203.97.78.43 and 203.97.78.44.

So you recommend these over the Paradise ones (203.96.152.4 and 203.96.152.12)? Are they the Clear ones?

edit: I get ~10ms for 203.96.152.*, and ~20ms for 203.97.78.*

olof

154 posts

Master Geek

Trusted

One NZ

#195748 13-Feb-2009 20:39

Screeb:
olof:
Ps. For optimum performance, we recommend that TelstraClear customers use the following DNS servers: 203.97.78.43 and 203.97.78.44.

So you recommend these over the Paradise ones (203.96.152.4 and 203.96.152.12)? Are they the Clear ones?

edit: I get ~10ms for 203.96.152.*, and ~20ms for 203.97.78.*

Screeb: most customers will have a lower latency to the 203.97.78.* servers. However, if your latency is lower to the Paradise 203.96.152.* ones you should stick to those. DNS server performance wise, they are similar and should both be good. The difference, is the latency depending on your access method and location.

Olof Olsson - my views are my own.

Sharesies

Trade NZ and US shares and funds with Sharesies (affiliate link).

PenultimateHop

637 posts

Ultimate Geek

Trusted

#195761 13-Feb-2009 21:30

olof:
The above is unfortunate, as OpenDNS is an _excellent_ service. Until this issue has been resolved, we recommend that TelstraClear customers do not use overseas DNS servers, like OpenDNS, as this will cause issues with geographically load-balanced sites.

I have offered to workwith OpenDNS to come up with a solution.

There is of course some options TCL could look at:

a) Don't mangle customer source IP when the traffic originates from the cache. This would resolve a number of asymmetry headaches I've seen with the TCL caches (egress from TCL is international, return path is domestic).
a1) Don't mangle customer source IP when the traffic is going to Akamai or other geo CDNs.
b) Don't divert traffic to the cache when it's going to an Akamai address (nb. this assumes you have a DPI engine that's doing the inspection for cache diversion). In fact, this is what Akamai prefers...
c) Ditch transparent caching altogether due to the extremely high administrative overhead and low payback on byte caches ;).
d) Pass all return traffic through the cache path, be it domestic or international.

olof

154 posts

Master Geek

Trusted

One NZ

#195769 13-Feb-2009 22:31

PenultimateHop:
olof:
The above is unfortunate, as OpenDNS is an _excellent_ service. Until this issue has been resolved, we recommend that TelstraClear customers do not use overseas DNS servers, like OpenDNS, as this will cause issues with geographically load-balanced sites.

I have offered to workwith OpenDNS to come up with a solution.

There is of course some options TCL could look at:

a) Don't mangle customer source IP when the traffic originates from the cache. This would resolve a number of asymmetry headaches I've seen with the TCL caches (egress from TCL is international, return path is domestic).
a1) Don't mangle customer source IP when the traffic is going to Akamai or other geo CDNs.
b) Don't divert traffic to the cache when it's going to an Akamai address (nb. this assumes you have a DPI engine that's doing the inspection for cache diversion). In fact, this is what Akamai prefers...
c) Ditch transparent caching altogether due to the extremely high administrative overhead and low payback on byte caches ;).
d) Pass all return traffic through the cache path, be it domestic or international.

PenultimateHop: We have been (and are) considering most of those. Suffice it to say, that there are pros and cons with all options. (However, I strongly disagree with you on the low payback. The payback is very signifcant from two perspectives: 1) improved customer HTTP performance and 2) increased effectiveness of international transit capacity. It is pretty simple. If we did not think that the caches were good both for our customers and for providing cost-effective broadband, we would not have them in the first place.)

Olof Olsson - my views are my own.

PenultimateHop

637 posts

Ultimate Geek

Trusted

#196487 18-Feb-2009 12:49

olof: PenultimateHop: We have been (and are) considering most of those. Suffice it to say, that there are pros and cons with all options. (However, I strongly disagree with you on the low payback. The payback is very signifcant from two perspectives: 1) improved customer HTTP performance and 2) increased effectiveness of international transit capacity. It is pretty simple. If we did not think that the caches were good both for our customers and for providing cost-effective broadband, we would not have them in the first place.)

Hi Olof,

I don't doubt you that TCL have crunched the numbers and consider the caches cost-effective. I definitely understand the benefits to caching, particularly around the user-experience/perception cases, with window size rewriting and latency reductions (especially as it allows the last mile to fill much more nicely). However these days I really am not sure about the actual byte savings on transit links[1] and the business case for it.

I've recently been looking at a project which wants to pass 5-6Gbps of HTTP traffic through transparent caches, and it seems to me that once the OpEx costs of operating the caches are included, it's cheaper to just buy more bandwidth than infrastructure to support 10G (or multiple G capable) caches. The headaches that come with breaking applications, poorly coded services, and having to deal with the munging/vs. non-munging aspects of caching that break end-to-end connectivity just seem too much of a hassle now, especially when you're talking of a 15% or less byte saving on transit links. Especially as more and more content becomes less cachable (talk to the Squid guys about what YouTube could be doing, but isn't).

I am in favor of P2P caching though -- this seems to make a lot more economic sense and has delivered big transit bandwidth savings to the networks I have seen it deployed in.

[1] I used to be a big fan of transparent HTTP caching and have deployed it in many networks, both in a munging and non-munging fashion. In the last 2 years I have really struggled with the business case, and now just can't see it stacking up any more.

Behodar

10508 posts

Uber Geek

Trusted

Lifetime subscriber

#197726 24-Feb-2009 12:44

From the blog:

2) Errors with automatic logins
Issues with logins to web sites such as Twitter not staying persistent.

Aha! It's good to know that it's not just me. I haven't seen this issue in a couple of weeks but I'll keep my eye open for if it happens again (I'm using TCL DNS on PDQ ASDL).

magu

Professional yak shaver
1599 posts

Uber Geek

Trusted

BitSignal

Lifetime subscriber

#197730 24-Feb-2009 13:01

Behodar: From the blog:

2) Errors with automatic logins
Issues with logins to web sites such as Twitter not staying persistent.

Aha! It's good to know that it's not just me. I haven't seen this issue in a couple of weeks but I'll keep my eye open for if it happens again (I'm using TCL DNS on PDQ ASDL).

So it's not just me? WOW.

"Roads? Where we're going, we don't need roads." - Doc Emmet Brown

Zach

167 posts

Master Geek

#201805 18-Mar-2009 03:47

Thanks for this information. I have updated my DNS settings all to 203.97.78.43/44. Before, I didn't realise that the computers that were slow or never connected to sites (like facebook) were caused by OpenDNS. Updating to 203.97.78.43/44 from 203.96.152.4/12 also fixed issues I had with sites like escapistmagazine (rarely connected) it seems.

mjb

996 posts

Ultimate Geek

Trusted

#201848 18-Mar-2009 09:02

freitasm: Here is a screenshot from an error just now, while accssing a website. Note it's not the ads, but the whole site:

That's interesting.. what browser? that doesn't look like a typical IE/FF error page. I'm guessing that's the proxy giving you the html. packet trace time. :)

Oh... heh, I wonder if telstra are blocking ICMP. (needs fragment/MTU/etc/etc).

contentsofsignaturemaysettleduringshipping

freitasm

BDFL - Memuneh
79288 posts

Uber Geek

Administrator

ID Verified

Trusted

Geekzone

Lifetime subscriber

#201863 18-Mar-2009 09:40

Did you read the rest of the thread? This was a transparent proxy problem, TelstraClear was on the ball, a patch was applied, all fixed now.

1 | 2 | 3