TC/Vodafone Stuff Problems

Forums › One NZ (including Farmside) › TC/Vodafone Stuff Problems

timmmay

20858 posts

Uber Geek
+1 received by user: 5350

Trusted

Lifetime subscriber

#129415 14-Sep-2013 22:48

It's been like this all day... Stuff loads eventually, but it takes 45 seconds. Not kidding, 45 seconds, waiting for static2.stuff.co.nz

Here's the traceroute, going via Tokyo it seems. This should be a 40ms ping, not 280ms. Is this another Vodafone routing issue, or is it a stuff issue?

tracert static2.stuff.co.nz

Tracing route to a1784.g.akamai.net [125.56.218.19]
over a maximum of 30 hops:

1 <1 ms 1 ms 1 ms 192.168.1.1
2 * * * Request timed out.
3 11 ms 10 ms 9 ms lo0.internet.ivpn.pexx.telstraclear.net
4 19 ms 20 ms 21 ms xx-g-0-0-0.telstraclear.net
5 19 ms 20 ms 21 ms ge-0-2-0-1.xcore1.acld.telstraclear.net
6 39 ms 20 ms 33 ms unknown.telstraglobal.net [134.159.174.37]
7 149 ms 147 ms 146 ms i-0-0-4-0.tlot-core01.bx.telstraglobal.net [202.84.142.118]
8 149 ms 145 ms 147 ms i-0-4-0-0.tlot02.bi.telstraglobal.net [202.84.251.238]
9 144 ms 144 ms 147 ms 208.173.53.141
10 152 ms 152 ms 151 ms 208.175.201.14
11 153 ms 153 ms 173 ms ae-6.r21.lsanca03.us.bb.gin.ntt.net [129.250.5.69]
12 300 ms 277 ms 280 ms as-1.r21.tokyjp01.jp.bb.gin.ntt.net [129.250.3.146]
13 253 ms 254 ms 266 ms ae-3.r24.tokyjp05.jp.bb.gin.ntt.net [129.250.6.188]
14 299 ms 291 ms 267 ms ae-3.r00.tokyjp03.jp.bb.gin.ntt.net [129.250.4.233]
15 270 ms 269 ms 268 ms ae-0.akamai.tokyjp03.jp.bb.gin.ntt.net [61.120.145.202]
16 276 ms 280 ms 283 ms a125-56.218-19.deploy.akamaitechnologies.com [125.56.218.19]

1 | 2 | 3

20858 posts

Uber Geek
+1 received by user: 5350

Trusted

Lifetime subscriber

#895799 15-Sep-2013 10:54

It was still doing it this morning, but it's come right in the past half hour. I wonder if it's a Vodafone or a stuff issue.

Here are a couple of interesting speed test results on stuff.co.nz
From NZ server: Test results.
From USA server: Test results.

tracert cdn.gigya.com

Tracing route to a221.g.akamai.net [125.56.201.123]
over a maximum of 30 hops:

1 <1 ms 1 ms <1 ms 192.168.1.1
2 * * * Request timed out.
3 11 ms 10 ms 10 ms lo0.internet.ivpn.pe25.telstraclear.net [218.101.61.124]
4 19 ms 20 ms 18 ms ie2-g-0-0-0.telstraclear.net [203.98.50.2]
5 20 ms 19 ms 18 ms ge-0-2-0-1.xcore1.acld.telstraclear.net [203.98.50.251]
6 22 ms 20 ms 29 ms unknown.telstraglobal.net [134.159.174.37]
7 147 ms 147 ms 147 ms i-0-0-1-1.1wlt-core01.bx.telstraglobal.net [202.84.223.85]
8 145 ms 145 ms 148 ms i-0-4-0-0.tlot02.bi.telstraglobal.net [202.84.251.238]
9 145 ms 151 ms 144 ms 208.175.201.5
10 153 ms 151 ms 153 ms 208.175.201.14
11 150 ms 151 ms 151 ms ae-8.r20.lsanca03.us.bb.gin.ntt.net [129.250.5.1]
12 260 ms 279 ms 267 ms ae-7.r20.tokyjp05.jp.bb.gin.ntt.net [129.250.3.15]
13 260 ms 259 ms 275 ms ae-19.r25.tokyjp05.jp.bb.gin.ntt.net [129.250.6.213]
14 289 ms 289 ms 298 ms ae-2.r00.tokyjp03.jp.bb.gin.ntt.net [129.250.2.5]
15 260 ms 268 ms 260 ms ae-0.akamai.tokyjp03.jp.bb.gin.ntt.net [61.120.145.202]
16 289 ms 287 ms 282 ms a125-56.201-123.deploy.akamaitechnologies.com [125.56.201.123]

tracert static2.stuff.co.nz

Tracing route to a1784.g.akamai.net [125.56.218.19]
over a maximum of 30 hops:

1 <1 ms 1 ms <1 ms 192.168.1.1
2 * * * Request timed out.
3 324 ms 53 ms 269 ms lo0.internet.ivpn.pe25.telstraclear.net [218.101.61.124]
4 20 ms 18 ms 34 ms ie2-g-0-0-0.telstraclear.net [203.98.50.2]
5 28 ms 21 ms 19 ms ge-0-2-0-1.xcore1.acld.telstraclear.net [203.98.50.251]
6 22 ms 21 ms 21 ms unknown.telstraglobal.net [134.159.174.37]
7 147 ms 147 ms 147 ms i-0-0-4-0.tlot-core01.bx.telstraglobal.net [202.84.142.118]
8 144 ms 145 ms 145 ms i-0-4-0-0.tlot02.bi.telstraglobal.net [202.84.251.238]
9 145 ms 146 ms 144 ms 208.175.201.5
10 150 ms 149 ms 150 ms 208.175.201.14
11 149 ms 157 ms * ae-6.r21.lsanca03.us.bb.gin.ntt.net [129.250.5.69]
12 266 ms 283 ms 267 ms ae-7.r20.tokyjp05.jp.bb.gin.ntt.net [129.250.3.15]
13 284 ms 284 ms 257 ms ae-19.r24.tokyjp05.jp.bb.gin.ntt.net [129.250.6.209]
14 263 ms 268 ms 269 ms ae-3.r00.tokyjp03.jp.bb.gin.ntt.net [129.250.4.233]
15 261 ms 270 ms 270 ms ae-0.akamai.tokyjp03.jp.bb.gin.ntt.net [61.120.145.202]
16 284 ms 286 ms 273 ms a125-56.218-19.deploy.akamaitechnologies.com [125.56.218.19]

Kraven

738 posts

Ultimate Geek
+1 received by user: 190

#895871 15-Sep-2013 13:56

What DNS servers are you using? For Akamai CDN to work correctly you need to be using Vodafone's DNS servers.

timmmay

20858 posts

Uber Geek
+1 received by user: 5350

Trusted

Lifetime subscriber

#895889 15-Sep-2013 14:48

DNS1: 203.97.78.43
DNS2: 203.97.78.44
DNS3: 8.8.8.8

Those are the ones Mauricio recommended in the TC DNS thread.

Sounddude

I fix stuff!
1935 posts

Uber Geek
+1 received by user: 640

Trusted

2degrees

Lifetime subscriber

#895952 15-Sep-2013 16:54

Having 8.8.8.8 in your DNS could cause odd experiences when it comes to CDN's.

Whats probably happened is Akamai couldn't tell where you were located based on a DNS going to 8.8.8.8 (for some reason) and your akamai traffic came from some random akamai server.

I would recommended not using 8.8.8.8 as a resolver.

timmmay

20858 posts

Uber Geek
+1 received by user: 5350

Trusted

Lifetime subscriber

#895958 15-Sep-2013 17:11

It's the third resolver, so long as the TC name servers work it'll never get used, right? It's in my router, which is running DD-WRT.

Sounddude

I fix stuff!
1935 posts

Uber Geek
+1 received by user: 640

Trusted

2degrees

Lifetime subscriber

#895978 15-Sep-2013 17:43

timmmay: It's the third resolver, so long as the TC name servers work it'll never get used, right? It's in my router, which is running DD-WRT.

Depends on the software. It could be round robin'ing or using 1 until it fails then moves to the next, so eventually you get stuck on 8.8.8.8 until it fails and it moves back to #1.

Apple TV

Stream your favourite shows now on Apple TV (affiliate link).

mercutio

1392 posts

Uber Geek
+1 received by user: 134

#895979 15-Sep-2013 17:56

really it's stuff to blame for using akamai.

cdn's like cachefly don't have these issues, and have much better average performance, even to NZ when their nearest node is in sydney.

the pages won't necessarily load faster in nz, as akamai doesn't have dns servers in new zealand, and only really works well for very high volume web sites as every single akamai node has to cache the resource individually to get good performance.

anyway, you could in theory complain to akamai, as even if using 8.8.8.8 it should pass on location data of the user.

that said, 45 seconds is nowhere near what you should expect for 240 msec ping. bbc is higher ping away than that, and loads in about 4 seconds uncached, and about 2 seconds cached. (from memory) there most likely is some kind of dns or timeout issue happening. maybe a resource that can't load at all. maybe try pulling up the file on static2.stuff.co.nz individually and see how long it takes.

like from the uk, the first image i see on main page is 30k big and takes 1.1 second to load:

curl -O http://static2.stuff.co.nz/1379223534/138/9168138.jpg 0.00s user 0.01s system 1% cpu 1.162 total

mercutio

1392 posts

Uber Geek
+1 received by user: 134

#895983 15-Sep-2013 18:08

Sounddude:
timmmay: It's the third resolver, so long as the TC name servers work it'll never get used, right? It's in my router, which is running DD-WRT.

Depends on the software. It could be round robin'ing or using 1 until it fails then moves to the next, so eventually you get stuck on 8.8.8.8 until it fails and it moves back to #1.

it's likely to be dnsmasq, which can round robin a bit. :(

it can also parallel request, which is probably actually safer for this problem, but it seems that it is an issue with 8.8.8.8 and akamai in this instance.

unfortunately if he was to set both the top two dns servers on their own, he could hit problems like how orcon's primary dns servers both went down at once today. and i'd probably recommend setting 203.96.152.4 in the mix instead of 8.8.8.8.

it's likely to perform worse than the two normal clear dns servers, but it's still clear ip space and will still be regionalised as NZ -- it is the legacy paradise name server that most cable users are still using as was configured statically, and so isn't likely to stop working any time soon,

and is in wellington rather than anycast/akld (i'm not sure which clear is doing, but it's akld from akld)

you would probably do ok using a vodafone dns server as well, but i'm not sure if they're allowed/blocked from legacy clear users or not.

timmmay

20858 posts

Uber Geek
+1 received by user: 5350

Trusted

Lifetime subscriber

#895990 15-Sep-2013 18:18

Thanks guys, I've removed Google's DNS server and I've added in the other clear DNS server. I'll report back if I see anything different. If what's said above is correct it could explain why some websites really have problems at times - ebay and amazon for example.

mercutio

1392 posts

Uber Geek
+1 received by user: 134

#895991 15-Sep-2013 18:21

i just realised the most likely reason for a severe degradation rather than a minor delay is that you hit telstraclear's bug with their proxy servers, where if you go to an international site, and it returns a NZ IP, then the traffic is blackholed, and you can't reach it.

this issue has been going on for years, and it's negatively effected many users and sites over time. i hope with the vodafone takeover that this issue is resolved.

mercutio

1392 posts

Uber Geek
+1 received by user: 134

#895993 15-Sep-2013 18:23

timmmay: Thanks guys, I've removed Google's DNS server and I've added in the other clear DNS server. I'll report back if I see anything different. If what's said above is correct it could explain why some websites really have problems at times - ebay and amazon for example.

you can also use opera turbo, hola unblocker, or google's new proxy service to work around these issues. it's telstraclear transparent proxy issues, which effects http sites but not https.

Quic Broadband

Move to New Zealand's best fibre broadband service (affiliate link). Free setup code: R587125ERQ6VE. Note that to use Quic Broadband you must be comfortable with configuring your own router.

mercutio

1392 posts

Uber Geek
+1 received by user: 134

#895994 15-Sep-2013 18:25

timmmay: Thanks guys, I've removed Google's DNS server and I've added in the other clear DNS server. I'll report back if I see anything different. If what's said above is correct it could explain why some websites really have problems at times - ebay and amazon for example.

ebay and amazon have both had a few issues this year in general as well. if ebay is showing some error page it's unrelated. amazon has had issues as recently as last month:

http://gigaom.com/2013/08/19/amazons-u-s-front-end-web-site-is-down/

sbiddle

30853 posts

Uber Geek
+1 received by user: 9996

Retired Mod

Trusted

Biddle Corp

Lifetime subscriber

#895997 15-Sep-2013 18:33

A nslookup on 8.8.8.8 on my cable connection points to 125.56.201.121/123 whereas using the TCL DNS servers points to two different 203.97.86.235/200 which are both local.

Your router is definitely using Google DNS to get those results.

yitz

2238 posts

Uber Geek
+1 received by user: 594

#895998 15-Sep-2013 18:37

mercutio: i just realised the most likely reason for a severe degradation rather than a minor delay is that you hit telstraclear's bug with their proxy servers, where if you go to an international site, and it returns a NZ IP, then the traffic is blackholed, and you can't reach it.

Asymmetric routing is the term you're looking for

mercutio

1392 posts

Uber Geek
+1 received by user: 134

#895999 15-Sep-2013 18:39

yitz:
mercutio: i just realised the most likely reason for a severe degradation rather than a minor delay is that you hit telstraclear's bug with their proxy servers, where if you go to an international site, and it returns a NZ IP, then the traffic is blackholed, and you can't reach it.
Asymmetric routing is the term you're looking for

<rant>
i was hoping to try to convey it in a general way. aymmetric routing is a normal part of the internet, and doesn't really explain the weakness of their segregation.

it's kind of like having a passport stamp when you leave new zealand unless you go to australia, and a passport stamp when you come back from anywhere but australia. but if you come back from australia they don't stamp the passport, but they won't let you back in the country because you didn't have a return passport stamp, but you had one when you left.

actually maybe it'd be clearer, if you had a passport stamp when coming out or leaving from auckland, but not from wellington, but some international flights terminate in wellington rather than auckland. so normally if returning to auckland, your flight would leave auckland and come back on auckland, but it could be rerouted to wellington if there is hazardous landing conditions in auckland, or a flight problem to auckland.

most international traffic is assymetric, and a significant amount of national traffic is assymetric, but as long as the path is advertised to telecom or telstraclear at the time of making the request, then traffic to the website will work. but if that connection is severed and it falls back on international routes, and it doesn't take an international path both ways (such as when people use vocus international plus domestic peering this can happen) but if dns load balances between two locations one with paid domestic transit and the other without, or one in nz and one overseas then the site will invariably fail unless the load balancing is very consistent such as with nzherald.

anyway, it's easy to create the problem, you just make a web site that has multiple ip's, one in nz, one overseas, and and ip addresses for both of them to dns. then it'll often fail on telstraclear, and work on other isp's.

</rant>

telstraclear's network is broken for http web sites, and works most of the time. but will not work reliably with some less common situations, in the event of partial failures. if web sites use https instead they can bypass telstraclear's transparent proxy. and if they always use US routing or paid domestic peeering consistently for the same hostname they will work. anycast cdn's are more reliable than smart geoip based dns solutions.

1 | 2 | 3