Geekzone: technology news, blogs, forums
Guest
Welcome Guest.
You haven't logged in yet. If you don't have an account you can register now.


View this topic in a long page with up to 500 replies per page Create new topic
1 | 2 | 3
sbiddle
30853 posts

Uber Geek

Retired Mod
Trusted
Biddle Corp
Lifetime subscriber

  #192385 27-Jan-2009 21:21
Send private message

I find theage.com.au regularly has all sorts of issues when accessing it via TCL. Sometimes the whole page won't load and plenty of times I get the same timeouts on some of the ads. It's a real pain.

Affiliate link
 
 
 

Affiliate link: Find your next Lenovo laptop, desktop, workstation or tablet now.
freitasm

BDFL - Memuneh
74172 posts

Uber Geek

Administrator
ID Verified
Trusted
Geekzone
Lifetime subscriber

  #193749 3-Feb-2009 17:59
Send private message

Olof has posted a comment update in my blog.




Support Geekzone by subscribing, making a donation. or using one of our referral links: Sharesies | Goodsync  | Mighty Ape | Backblaze | Norton 360 | Lenovo laptops 

 

freitasm on Keybase | My technology disclosure

 

 

 

 

 

 


freitasm

BDFL - Memuneh
74172 posts

Uber Geek

Administrator
ID Verified
Trusted
Geekzone
Lifetime subscriber

  #194122 4-Feb-2009 22:05
Send private message

Folks, we've got an update - and I will post a new blog entry later.

As stated a patch is being released Thursday to address some proxy problems, and it could be the fix for the TCP errors we've seen.

As for OpenDNS, we have identified that this problem happens when TelstraClear users try to access content from Akamai servers.

There's a discussion on OpenDNS forums explaining how when a DNS request is made Akamai will return an IP that's closer to the user. When querying OpenDNS the IP will be the one closer to the node.

The client browser will request the content using the IP provided by openDNS and it seems that at certain point Akamai decides to send the packets back through the nearest node to the destination.

At this point a stateful firewall on their border will drop the packet since the request didn't come through that firewall.

If this is the case, the easiest fix would be for OpenDNS to have some nodes down here...

I will post this in my blog later. Big thanks to TelstraClear and Olof who worked on this even through the late hours...




Support Geekzone by subscribing, making a donation. or using one of our referral links: Sharesies | Goodsync  | Mighty Ape | Backblaze | Norton 360 | Lenovo laptops 

 

freitasm on Keybase | My technology disclosure

 

 

 

 

 

 




dub

dub
2 posts

Wannabe Geek
Inactive user


  #195286 11-Feb-2009 11:23
Send private message

If this is the case, the easiest fix would be for OpenDNS to have some nodes down here...


Unless they have a node at every ISP that hosts an Akamai deployment I expect the "problem" would remain, in some form.

Has your experience changed since the work they did on the caches last Thursday.

olof
154 posts

Master Geek

Trusted
Vodafone NZ

  #195721 13-Feb-2009 17:46
Send private message

All,

We have been able to isolate the issue to occur in the following circumstances as it relates to TelstraClear customers in New Zealand:

1. Customer is using a TelstraClear access method that is transparently cached
2. Customer has the OpenDNS US based DNS servers configured
3. Customer attempts to access site that uses geographic load-balancing (Akamized sites for example)

Under the above circumstances, the customer is unlikely to be able to access the web site.

The explanation and root cause of the issue is as follows. Let's use a fictional web site "shoerax.com", that uses Akamai to serve its content:

1) Customer's PC looks up "shoerax.com" against the OpenDNS DNS servers based in the US

2) shoerax.com uses Akamai, and Akamai DNS servers. Hence the customer's DNS lookup goes to the OpenDNS DNS servers in the US, and these servers recursively look up "shoerax.com" from the Akamai DNS servers. As Akamai uses geographic load-balancing they attempt to return the IPs of servers that "are close to the customer". Akamai use the source IP of the querying recursive DNS server (= OpenDNS US based servers in this instance) to determine what "close to the customer is". Normally this works fine, as most people use recursive DNS servers that are close to where they are. However, when NZ customers use US based DNS servers (like OpenDNS), things break down a little, as US based server IPs are returned. (And they are obviously not close to the customer!) Hence, the Akamai DNS returns IP addresses of US based Akamai servers for shoerax.com.

3) The customer's PC attempts to establish an HTTP connection with the US based Akamai servers. (Using the IP addresses returned by OpenDNS.)

4) The TelstraClear transparent caches intercept the HTTP traffic. The transparent caches look up shoerax.com against TelstraClear DNS servers. As these DNS servers are in NZ, Akamai correctly returns the IP addresses of NZ based Akamai servers.

5) The caches attempt to connect to the NZ Akamai servers to retrieve the web objects.

6) The NZ Akamai servers sends return packets to the customers PC's (as this is the source IP of the incoming traffic). This is where the problem is. The return traffic should have gone to the caches and not to the customer's PC.

7) Caches time out

8) Customer's browser times out

The TelstraClear caches are situated on the NZ side of our international circuits and only attempt to cache international traffic. They expect that egress international HTTP traffic has return traffic coming back on the international circuits.

The issue with OpenDNS and TelstraClear transparent caching in conjunction with geographically load-balanced sites, is that the traffic is initially sent internationally (due to lookups against US based OpenDNS servers) and then the caches sends the traffic back domestically (due to lookups against NZ based TelstraClear servers). Due to this asymmetry, return traffic from the target web site do not reach the caches.

The above is unfortunate, as OpenDNS is an _excellent_ service. Until this issue has been resolved, we recommend that TelstraClear customers do not use overseas DNS servers, like OpenDNS, as this will cause issues with geographically load-balanced sites.

I have offered to workwith OpenDNS to come up with a solution.

--Olof

Ps. For optimum performance, we recommend that TelstraClear customers use the following DNS servers: 203.97.78.43 and 203.97.78.44.

Screeb
687 posts

Ultimate Geek


  #195735 13-Feb-2009 19:04
Send private message

olof:
Ps. For optimum performance, we recommend that TelstraClear customers use the following DNS servers: 203.97.78.43 and 203.97.78.44.


So you recommend these over the Paradise ones (203.96.152.4 and 203.96.152.12)? Are they the Clear ones?

edit: I get ~10ms for 203.96.152.*, and ~20ms for 203.97.78.*

olof
154 posts

Master Geek

Trusted
Vodafone NZ

  #195748 13-Feb-2009 20:39
Send private message

Screeb:
olof:
Ps. For optimum performance, we recommend that TelstraClear customers use the following DNS servers: 203.97.78.43 and 203.97.78.44.


So you recommend these over the Paradise ones (203.96.152.4 and 203.96.152.12)? Are they the Clear ones?

edit: I get ~10ms for 203.96.152.*, and ~20ms for 203.97.78.*


Screeb: most customers will have a lower latency to the 203.97.78.* servers. However, if your latency is lower to the Paradise 203.96.152.* ones you should stick to those. DNS server performance wise, they are similar and should both be good. The difference, is the latency depending on your access method and location.




Olof Olsson - my views are my own. 




PenultimateHop
637 posts

Ultimate Geek

Trusted

  #195761 13-Feb-2009 21:30
Send private message

olof:
The above is unfortunate, as OpenDNS is an _excellent_ service. Until this issue has been resolved, we recommend that TelstraClear customers do not use overseas DNS servers, like OpenDNS, as this will cause issues with geographically load-balanced sites.

I have offered to workwith OpenDNS to come up with a solution.

There is of course some options TCL could look at:

a) Don't mangle customer source IP when the traffic originates from the cache.  This would resolve a number of asymmetry headaches I've seen with the TCL caches (egress from TCL is international, return path is domestic).
 a1) Don't mangle customer source IP when the traffic is going to Akamai or other geo CDNs.
b) Don't divert traffic to the cache when it's going to an Akamai address (nb. this assumes you have a DPI engine that's doing the inspection for cache diversion).  In fact, this is what Akamai prefers...
c) Ditch transparent caching altogether due to the extremely high administrative overhead and low payback on byte caches ;).
d) Pass all return traffic through the cache path, be it domestic or international.

olof
154 posts

Master Geek

Trusted
Vodafone NZ

  #195769 13-Feb-2009 22:31
Send private message

PenultimateHop:
olof:
The above is unfortunate, as OpenDNS is an _excellent_ service. Until this issue has been resolved, we recommend that TelstraClear customers do not use overseas DNS servers, like OpenDNS, as this will cause issues with geographically load-balanced sites.

I have offered to workwith OpenDNS to come up with a solution.

There is of course some options TCL could look at:

a) Don't mangle customer source IP when the traffic originates from the cache.  This would resolve a number of asymmetry headaches I've seen with the TCL caches (egress from TCL is international, return path is domestic).
 a1) Don't mangle customer source IP when the traffic is going to Akamai or other geo CDNs.
b) Don't divert traffic to the cache when it's going to an Akamai address (nb. this assumes you have a DPI engine that's doing the inspection for cache diversion).  In fact, this is what Akamai prefers...
c) Ditch transparent caching altogether due to the extremely high administrative overhead and low payback on byte caches ;).
d) Pass all return traffic through the cache path, be it domestic or international.


PenultimateHop: We have been (and are) considering most of those. Suffice it to say, that there are pros and cons with all options. (However, I strongly disagree with you on the low payback. The payback is very signifcant from two perspectives: 1) improved customer HTTP performance and 2) increased effectiveness of international transit capacity. It is pretty simple. If we did not think that the caches were good both for our customers and for providing cost-effective broadband, we would not have them in the first place.)





Olof Olsson - my views are my own. 


PenultimateHop
637 posts

Ultimate Geek

Trusted

  #196487 18-Feb-2009 12:49
Send private message

olof: PenultimateHop: We have been (and are) considering most of those. Suffice it to say, that there are pros and cons with all options. (However, I strongly disagree with you on the low payback. The payback is very signifcant from two perspectives: 1) improved customer HTTP performance and 2) increased effectiveness of international transit capacity. It is pretty simple. If we did not think that the caches were good both for our customers and for providing cost-effective broadband, we would not have them in the first place.)

Hi Olof,

I don't doubt you that TCL have crunched the numbers and consider the caches cost-effective.  I definitely understand the benefits to caching, particularly around the user-experience/perception cases, with window size rewriting and latency reductions (especially as it allows the last mile to fill much more nicely).  However these days I really am not sure about the actual byte savings on transit links[1] and the business case for it. 

I've recently been looking at a project which wants to pass 5-6Gbps of HTTP traffic through transparent caches, and it seems to me that once the OpEx costs of operating the caches are included, it's cheaper to just buy more bandwidth than infrastructure to support 10G (or multiple G capable) caches.  The headaches that come with breaking applications, poorly coded services, and having to deal with the munging/vs. non-munging aspects of caching that break end-to-end connectivity just seem too much of a hassle now, especially when you're talking of a 15% or less byte saving on transit links.  Especially as more and more content becomes less cachable (talk to the Squid guys about what YouTube could be doing, but isn't).

I am in favor of P2P caching though -- this seems to make a lot more economic sense and has delivered big transit bandwidth savings to the networks I have seen it deployed in.

[1] I used to be a big fan of transparent HTTP caching and have deployed it in many networks, both in a munging and non-munging fashion.  In the last 2 years I have really struggled with the business case, and now just can't see it stacking up any more.

Behodar
8371 posts

Uber Geek

Trusted
Lifetime subscriber

  #197726 24-Feb-2009 12:44
Send private message

From the blog:

2) Errors with automatic logins
Issues with logins to web sites such as Twitter not staying persistent.


Aha! It's good to know that it's not just me. I haven't seen this issue in a couple of weeks but I'll keep my eye open for if it happens again (I'm using TCL DNS on PDQ ASDL).

magu
Professional yak shaver
1599 posts

Uber Geek

Trusted
BitSignal
Lifetime subscriber

  #197730 24-Feb-2009 13:01
Send private message

Behodar: From the blog:

2) Errors with automatic logins
Issues with logins to web sites such as Twitter not staying persistent.


Aha! It's good to know that it's not just me. I haven't seen this issue in a couple of weeks but I'll keep my eye open for if it happens again (I'm using TCL DNS on PDQ ASDL).

So it's not just me? WOW.




"Roads? Where we're going, we don't need roads." - Doc Emmet Brown

Zach
167 posts

Master Geek


  #201805 18-Mar-2009 03:47
Send private message

Thanks for this information. I have updated my DNS settings all to 203.97.78.43/44. Before, I didn't realise that the computers that were slow or never connected to sites (like facebook) were caused by OpenDNS. Updating to 203.97.78.43/44 from 203.96.152.4/12 also fixed issues I had with sites like escapistmagazine (rarely connected) it seems.

mjb

mjb
927 posts

Ultimate Geek

Trusted

  #201848 18-Mar-2009 09:02
Send private message

freitasm: Here is a screenshot from an error just now, while accssing a website. Note it's not the ads, but the whole site:




That's interesting.. what browser? that doesn't look like a typical IE/FF error page. I'm guessing that's the proxy giving you the html. packet trace time. :)

Oh... heh, I wonder if telstra are blocking ICMP. (needs fragment/MTU/etc/etc).




contentsofsignaturemaysettleduringshipping


freitasm

BDFL - Memuneh
74172 posts

Uber Geek

Administrator
ID Verified
Trusted
Geekzone
Lifetime subscriber

#201863 18-Mar-2009 09:40
Send private message

Did you read the rest of the thread? This was a transparent proxy problem, TelstraClear was on the ball, a patch was applied, all fixed now.




Support Geekzone by subscribing, making a donation. or using one of our referral links: Sharesies | Goodsync  | Mighty Ape | Backblaze | Norton 360 | Lenovo laptops 

 

freitasm on Keybase | My technology disclosure

 

 

 

 

 

 


1 | 2 | 3
View this topic in a long page with up to 500 replies per page Create new topic





News and reviews »

Belkin Screenforce Tempered Glass Screen Protector and Bumper - Apple Watch
Posted 15-Aug-2022 17:20


Samsung Introducing Galaxy Z Flip4 and Galaxy Z Fold4
Posted 11-Aug-2022 01:00


Samsung Unveils Health Innovations with Galaxy Watch5 and Galaxy Watch5 Pro
Posted 11-Aug-2022 01:00


Google Bringing First Cloud Region to Aotearoa New Zealand
Posted 10-Aug-2022 08:51


ANZ To Move to FIS Modern Banking Platform
Posted 10-Aug-2022 08:28


GoPro Hero10 Black Review
Posted 8-Aug-2022 17:41


Amazon to Acquire iRobot
Posted 6-Aug-2022 11:41


Samsung x LIFE Picture Collection Brings Iconic Moments in History to The Frame
Posted 4-Aug-2022 17:04


Norton Consumer Cyber Safety Pulse Report: Phishing for New Bait on Social Media
Posted 4-Aug-2022 16:50


Microsoft Announces New Solutions for Threat Intelligence and Attack Surface Management
Posted 3-Aug-2022 21:54


Seagate Addresses Hyperscale Workloads with Enterprise-Class Nytro SSDs
Posted 3-Aug-2022 21:50


Visa Launching Eco-friendly Payment Solutions in New Zealand
Posted 3-Aug-2022 21:48


NCR Delivers Services to Run Bank of New Zealand ATM Network
Posted 30-Jul-2022 11:06


New HP Portfolio Supports New Era of Hybrid Work
Posted 28-Jul-2022 17:14


Harman Kardon Launches Citation MultiBeam 1100 Soundbar
Posted 28-Jul-2022 17:10









Geekzone Live »

Try automatic live updates from Geekzone directly in your browser, without refreshing the page, with Geekzone Live now.



Are you subscribed to our RSS feed? You can download the latest headlines and summaries from our stories directly to your computer or smartphone by using a feed reader.







GoodSync is the easiest file sync and backup for Windows and Mac