Geekzone: technology news, blogs, forums
Guest
Welcome Guest.
You haven't logged in yet. If you don't have an account you can register now.


View this topic in a long page with up to 500 replies per page Create new topic
1 | 2 | 3
27896 posts

Uber Geek
+1 received by user: 7383

Moderator
Trusted
Biddle Corp
Lifetime subscriber

  # 192385 27-Jan-2009 21:21
Send private message

I find theage.com.au regularly has all sorts of issues when accessing it via TCL. Sometimes the whole page won't load and plenty of times I get the same timeouts on some of the ads. It's a real pain.



BDFL - Memuneh
63630 posts

Uber Geek
+1 received by user: 14092

Administrator
Trusted
Geekzone
Lifetime subscriber

 
 
 
 




BDFL - Memuneh
63630 posts

Uber Geek
+1 received by user: 14092

Administrator
Trusted
Geekzone
Lifetime subscriber

  # 194122 4-Feb-2009 22:05
Send private message

Folks, we've got an update - and I will post a new blog entry later.

As stated a patch is being released Thursday to address some proxy problems, and it could be the fix for the TCP errors we've seen.

As for OpenDNS, we have identified that this problem happens when TelstraClear users try to access content from Akamai servers.

There's a discussion on OpenDNS forums explaining how when a DNS request is made Akamai will return an IP that's closer to the user. When querying OpenDNS the IP will be the one closer to the node.

The client browser will request the content using the IP provided by openDNS and it seems that at certain point Akamai decides to send the packets back through the nearest node to the destination.

At this point a stateful firewall on their border will drop the packet since the request didn't come through that firewall.

If this is the case, the easiest fix would be for OpenDNS to have some nodes down here...

I will post this in my blog later. Big thanks to TelstraClear and Olof who worked on this even through the late hours...




dub

2 posts

Wannabe Geek


  # 195286 11-Feb-2009 11:23
Send private message

If this is the case, the easiest fix would be for OpenDNS to have some nodes down here...


Unless they have a node at every ISP that hosts an Akamai deployment I expect the "problem" would remain, in some form.

Has your experience changed since the work they did on the caches last Thursday.

64 posts

Master Geek

Trusted
TelstraClear

  # 195721 13-Feb-2009 17:46
Send private message

All,

We have been able to isolate the issue to occur in the following circumstances as it relates to TelstraClear customers in New Zealand:

1. Customer is using a TelstraClear access method that is transparently cached
2. Customer has the OpenDNS US based DNS servers configured
3. Customer attempts to access site that uses geographic load-balancing (Akamized sites for example)

Under the above circumstances, the customer is unlikely to be able to access the web site.

The explanation and root cause of the issue is as follows. Let's use a fictional web site "shoerax.com", that uses Akamai to serve its content:

1) Customer's PC looks up "shoerax.com" against the OpenDNS DNS servers based in the US

2) shoerax.com uses Akamai, and Akamai DNS servers. Hence the customer's DNS lookup goes to the OpenDNS DNS servers in the US, and these servers recursively look up "shoerax.com" from the Akamai DNS servers. As Akamai uses geographic load-balancing they attempt to return the IPs of servers that "are close to the customer". Akamai use the source IP of the querying recursive DNS server (= OpenDNS US based servers in this instance) to determine what "close to the customer is". Normally this works fine, as most people use recursive DNS servers that are close to where they are. However, when NZ customers use US based DNS servers (like OpenDNS), things break down a little, as US based server IPs are returned. (And they are obviously not close to the customer!) Hence, the Akamai DNS returns IP addresses of US based Akamai servers for shoerax.com.

3) The customer's PC attempts to establish an HTTP connection with the US based Akamai servers. (Using the IP addresses returned by OpenDNS.)

4) The TelstraClear transparent caches intercept the HTTP traffic. The transparent caches look up shoerax.com against TelstraClear DNS servers. As these DNS servers are in NZ, Akamai correctly returns the IP addresses of NZ based Akamai servers.

5) The caches attempt to connect to the NZ Akamai servers to retrieve the web objects.

6) The NZ Akamai servers sends return packets to the customers PC's (as this is the source IP of the incoming traffic). This is where the problem is. The return traffic should have gone to the caches and not to the customer's PC.

7) Caches time out

8) Customer's browser times out

The TelstraClear caches are situated on the NZ side of our international circuits and only attempt to cache international traffic. They expect that egress international HTTP traffic has return traffic coming back on the international circuits.

The issue with OpenDNS and TelstraClear transparent caching in conjunction with geographically load-balanced sites, is that the traffic is initially sent internationally (due to lookups against US based OpenDNS servers) and then the caches sends the traffic back domestically (due to lookups against NZ based TelstraClear servers). Due to this asymmetry, return traffic from the target web site do not reach the caches.

The above is unfortunate, as OpenDNS is an _excellent_ service. Until this issue has been resolved, we recommend that TelstraClear customers do not use overseas DNS servers, like OpenDNS, as this will cause issues with geographically load-balanced sites.

I have offered to workwith OpenDNS to come up with a solution.

--Olof

Ps. For optimum performance, we recommend that TelstraClear customers use the following DNS servers: 203.97.78.43 and 203.97.78.44.

671 posts

Ultimate Geek
+1 received by user: 10


  # 195735 13-Feb-2009 19:04
Send private message

olof:
Ps. For optimum performance, we recommend that TelstraClear customers use the following DNS servers: 203.97.78.43 and 203.97.78.44.


So you recommend these over the Paradise ones (203.96.152.4 and 203.96.152.12)? Are they the Clear ones?

edit: I get ~10ms for 203.96.152.*, and ~20ms for 203.97.78.*

64 posts

Master Geek

Trusted
TelstraClear

  # 195748 13-Feb-2009 20:39
Send private message

Screeb:
olof:
Ps. For optimum performance, we recommend that TelstraClear customers use the following DNS servers: 203.97.78.43 and 203.97.78.44.


So you recommend these over the Paradise ones (203.96.152.4 and 203.96.152.12)? Are they the Clear ones?

edit: I get ~10ms for 203.96.152.*, and ~20ms for 203.97.78.*


Screeb: most customers will have a lower latency to the 203.97.78.* servers. However, if your latency is lower to the Paradise 203.96.152.* ones you should stick to those. DNS server performance wise, they are similar and should both be good. The difference, is the latency depending on your access method and location.




---
http://olofolsson.com/

 
 
 
 


637 posts

Ultimate Geek
+1 received by user: 2

Trusted

  # 195761 13-Feb-2009 21:30
Send private message

olof:
The above is unfortunate, as OpenDNS is an _excellent_ service. Until this issue has been resolved, we recommend that TelstraClear customers do not use overseas DNS servers, like OpenDNS, as this will cause issues with geographically load-balanced sites.

I have offered to workwith OpenDNS to come up with a solution.

There is of course some options TCL could look at:

a) Don't mangle customer source IP when the traffic originates from the cache.  This would resolve a number of asymmetry headaches I've seen with the TCL caches (egress from TCL is international, return path is domestic).
 a1) Don't mangle customer source IP when the traffic is going to Akamai or other geo CDNs.
b) Don't divert traffic to the cache when it's going to an Akamai address (nb. this assumes you have a DPI engine that's doing the inspection for cache diversion).  In fact, this is what Akamai prefers...
c) Ditch transparent caching altogether due to the extremely high administrative overhead and low payback on byte caches ;).
d) Pass all return traffic through the cache path, be it domestic or international.

64 posts

Master Geek

Trusted
TelstraClear

  # 195769 13-Feb-2009 22:31
Send private message

PenultimateHop:
olof:
The above is unfortunate, as OpenDNS is an _excellent_ service. Until this issue has been resolved, we recommend that TelstraClear customers do not use overseas DNS servers, like OpenDNS, as this will cause issues with geographically load-balanced sites.

I have offered to workwith OpenDNS to come up with a solution.

There is of course some options TCL could look at:

a) Don't mangle customer source IP when the traffic originates from the cache.  This would resolve a number of asymmetry headaches I've seen with the TCL caches (egress from TCL is international, return path is domestic).
 a1) Don't mangle customer source IP when the traffic is going to Akamai or other geo CDNs.
b) Don't divert traffic to the cache when it's going to an Akamai address (nb. this assumes you have a DPI engine that's doing the inspection for cache diversion).  In fact, this is what Akamai prefers...
c) Ditch transparent caching altogether due to the extremely high administrative overhead and low payback on byte caches ;).
d) Pass all return traffic through the cache path, be it domestic or international.


PenultimateHop: We have been (and are) considering most of those. Suffice it to say, that there are pros and cons with all options. (However, I strongly disagree with you on the low payback. The payback is very signifcant from two perspectives: 1) improved customer HTTP performance and 2) increased effectiveness of international transit capacity. It is pretty simple. If we did not think that the caches were good both for our customers and for providing cost-effective broadband, we would not have them in the first place.)





---
http://olofolsson.com/

637 posts

Ultimate Geek
+1 received by user: 2

Trusted

  # 196487 18-Feb-2009 12:49
Send private message

olof: PenultimateHop: We have been (and are) considering most of those. Suffice it to say, that there are pros and cons with all options. (However, I strongly disagree with you on the low payback. The payback is very signifcant from two perspectives: 1) improved customer HTTP performance and 2) increased effectiveness of international transit capacity. It is pretty simple. If we did not think that the caches were good both for our customers and for providing cost-effective broadband, we would not have them in the first place.)

Hi Olof,

I don't doubt you that TCL have crunched the numbers and consider the caches cost-effective.  I definitely understand the benefits to caching, particularly around the user-experience/perception cases, with window size rewriting and latency reductions (especially as it allows the last mile to fill much more nicely).  However these days I really am not sure about the actual byte savings on transit links[1] and the business case for it. 

I've recently been looking at a project which wants to pass 5-6Gbps of HTTP traffic through transparent caches, and it seems to me that once the OpEx costs of operating the caches are included, it's cheaper to just buy more bandwidth than infrastructure to support 10G (or multiple G capable) caches.  The headaches that come with breaking applications, poorly coded services, and having to deal with the munging/vs. non-munging aspects of caching that break end-to-end connectivity just seem too much of a hassle now, especially when you're talking of a 15% or less byte saving on transit links.  Especially as more and more content becomes less cachable (talk to the Squid guys about what YouTube could be doing, but isn't).

I am in favor of P2P caching though -- this seems to make a lot more economic sense and has delivered big transit bandwidth savings to the networks I have seen it deployed in.

[1] I used to be a big fan of transparent HTTP caching and have deployed it in many networks, both in a munging and non-munging fashion.  In the last 2 years I have really struggled with the business case, and now just can't see it stacking up any more.

6617 posts

Uber Geek
+1 received by user: 1318

Trusted
Lifetime subscriber

  # 197726 24-Feb-2009 12:44
Send private message

From the blog:

2) Errors with automatic logins
Issues with logins to web sites such as Twitter not staying persistent.


Aha! It's good to know that it's not just me. I haven't seen this issue in a couple of weeks but I'll keep my eye open for if it happens again (I'm using TCL DNS on PDQ ASDL).

Professional yak shaver
1599 posts

Uber Geek
+1 received by user: 8

Trusted
BitSignal
Lifetime subscriber

  # 197730 24-Feb-2009 13:01
Send private message

Behodar: From the blog:

2) Errors with automatic logins
Issues with logins to web sites such as Twitter not staying persistent.


Aha! It's good to know that it's not just me. I haven't seen this issue in a couple of weeks but I'll keep my eye open for if it happens again (I'm using TCL DNS on PDQ ASDL).

So it's not just me? WOW.




"Roads? Where we're going, we don't need roads." - Doc Emmet Brown

167 posts

Master Geek


  # 201805 18-Mar-2009 03:47
Send private message

Thanks for this information. I have updated my DNS settings all to 203.97.78.43/44. Before, I didn't realise that the computers that were slow or never connected to sites (like facebook) were caused by OpenDNS. Updating to 203.97.78.43/44 from 203.96.152.4/12 also fixed issues I had with sites like escapistmagazine (rarely connected) it seems.

mjb

923 posts

Ultimate Geek
+1 received by user: 21

Trusted

  # 201848 18-Mar-2009 09:02
Send private message

freitasm: Here is a screenshot from an error just now, while accssing a website. Note it's not the ads, but the whole site:




That's interesting.. what browser? that doesn't look like a typical IE/FF error page. I'm guessing that's the proxy giving you the html. packet trace time. :)

Oh... heh, I wonder if telstra are blocking ICMP. (needs fragment/MTU/etc/etc).




contentsofsignaturemaysettleduringshipping




BDFL - Memuneh
63630 posts

Uber Geek
+1 received by user: 14092

Administrator
Trusted
Geekzone
Lifetime subscriber

# 201863 18-Mar-2009 09:40
Send private message

Did you read the rest of the thread? This was a transparent proxy problem, TelstraClear was on the ball, a patch was applied, all fixed now.




1 | 2 | 3
View this topic in a long page with up to 500 replies per page Create new topic



Twitter and LinkedIn »



Follow us to receive Twitter updates when new discussions are posted in our forums:



Follow us to receive Twitter updates when news items and blogs are posted in our frontpage:



Follow us to receive Twitter updates when tech item prices are listed in our price comparison site:





News »

Xero announces new smarter tools, push into the North American market
Posted 19-Jun-2019 17:20


New report by Unisys shows New Zealanders want action by social platform companies and police to monitor social media sites
Posted 19-Jun-2019 17:09


ASB adds Google Pay option to contactless payments
Posted 19-Jun-2019 17:05


New Zealand PC Market declines on the back of high channel inventory, IDC reports
Posted 18-Jun-2019 17:35


Air New Zealand uses drones to inspect aircraft
Posted 17-Jun-2019 15:39


TCL Electronics launches its first-ever 8K TV
Posted 17-Jun-2019 15:18


E-scooter share scheme launches in Wellington
Posted 17-Jun-2019 12:34


Anyone can broadcast with Kordia Pop Up TV
Posted 13-Jun-2019 10:51


Volvo and Uber present production vehicle ready for self-driving
Posted 13-Jun-2019 10:47


100,000 customers connected to fibre broadband network through Enable
Posted 13-Jun-2019 10:35


5G uptake even faster than expected
Posted 12-Jun-2019 10:01


Xbox showcases 60 anticipated games
Posted 10-Jun-2019 20:24


Trend Micro Turns Public Hotspots into Secure Networks with WiFi Protection for Mobile Devices
Posted 5-Jun-2019 13:24


Bold UK spinoff for beauty software company Flossie
Posted 2-Jun-2019 14:10


Amazon Introduces Echo Show 5
Posted 1-Jun-2019 15:32



Geekzone Live »

Try automatic live updates from Geekzone directly in your browser, without refreshing the page, with Geekzone Live now.


Support Geekzone »

Our community of supporters help make Geekzone possible. Click the button below to join them.

Support Geezone on PressPatron



Are you subscribed to our RSS feed? You can download the latest headlines and summaries from our stories directly to your computer or smartphone by using a feed reader.

Alternatively, you can receive a daily email with Geekzone updates.