Vodafone improved browsing while TelstraClear DSL comes in last of top 5

Forums › New Zealand Broadband › Vodafone improved browsing while TelstraClear DSL comes in last of top 5

JohnButt

374 posts

Ultimate Geek

Trusted

#92827 7-Nov-2011 11:46

Latest broadband comparison statistics

October is the third month of data from our series of detailed broadband performance statistics. There were many improvements in performance in October over both August and September, but the greatest improvement was fromVodafone, moving from last to a position that would have seen them third in September, but for the competitive response from Orcon & Slingshot.

Orcon also improved to take over second position from Telecom with a second significant improvement month on month

This is also the first time TrueNet has had sufficient volunteers to report on TelstraClear DSL as well as TesltraClear Cable. While the Cable result continues to top the poll, the DSL result is well down the list at seventh.

File 596

The Vodafone improvement was the result of changes that affected almost all TrueNet Vodafone probes on the 7th October after some temporary improvements in September.

1 | 2 | 3

3892 posts

Uber Geek

#541972 7-Nov-2011 11:59

More nice work John! Top stuff man... this is very very cool stuff to see. :)

Promote New Zealand - Get yourself a .kiwi.nz domain name!!!

Check out mine - i.am.a.can.do.kiwi.nz - don@i.am.a.can.do.kiwi.nz

michaelmurfy

/dev/null
9150 posts

Uber Geek

Moderator

Trusted

Lifetime subscriber

#541987 7-Nov-2011 12:39

That's pretty awesome to see - all we need now is more probes at more locations.

JohnButt

374 posts

Ultimate Geek

Trusted

#541991 7-Nov-2011 12:45

michaelmurfy: That's pretty awesome to see - all we need now is more probes at more locations.

Actually a few more probes with some of the smaller ISPs would be good also, we have some but not enough with Actrix and Maxnet, but none yet from Woosh or Inspire, which I thought were of reasonable size.

Any others we should definitely cover?

Ragnor

8035 posts

Uber Geek

Trusted

#542018 7-Nov-2011 13:43

I think the default of interleaving on hurts Telecom's average quite a bit.. here's my Telecom connection at home for example:

johnr

19282 posts

Uber Geek
Inactive user

#542022 7-Nov-2011 13:49

Nice to see the hard work paying off

John

michaeln

220 posts

Master Geek

Trusted

#542042 7-Nov-2011 14:28

JohnButt: Latest broadband comparison statistics

This is also the first time TrueNet has had sufficient volunteers to report on TelstraClear DSL as well as TesltraClear Cable. While the Cable result continues to top the poll, the DSL result is well down the list at seventh.

Do you have an estimate of the error around these figures? I.e. "ISP A" Time to load x seconds [+/- y seconds].

Likewise, any statistical analysis around whether results are indeed statistically different?

JohnButt

374 posts

Ultimate Geek

Trusted

#542063 7-Nov-2011 15:09

michaeln:
JohnButt: Latest broadband comparison statistics

This is also the first time TrueNet has had sufficient volunteers to report on TelstraClear DSL as well as TesltraClear Cable. While the Cable result continues to top the poll, the DSL result is well down the list at seventh.

Do you have an estimate of the error around these figures? I.e. "ISP A" Time to load x seconds [+/- y seconds].

Likewise, any statistical analysis around whether results are indeed statistically different?

I am reluctant to provide everything. Let me assure you that the TelstraClear Cable results are very reliable, the worst cable measure has only one DSL probe with better results.

Note we use the Median probe performances per ISP, ie of the average performance for the month for each probe. Each probe has little variation, making the average reliable. That way we drop any outliers (if there are any - mine is) providing more reliable results.

jpollock

600 posts

Ultimate Geek

Trusted

#542445 8-Nov-2011 11:14

JohnButt:
michaeln:
JohnButt: Latest broadband comparison statistics

This is also the first time TrueNet has had sufficient volunteers to report on TelstraClear DSL as well as TesltraClear Cable. While the Cable result continues to top the poll, the DSL result is well down the list at seventh.

Do you have an estimate of the error around these figures? I.e. "ISP A" Time to load x seconds [+/- y seconds].

Likewise, any statistical analysis around whether results are indeed statistically different?

I am reluctant to provide everything. Let me assure you that the TelstraClear Cable results are very reliable, the worst cable measure has only one DSL probe with better results.

Note we use the Median probe performances per ISP, ie of the average performance for the month for each probe. Each probe has little variation, making the average reliable. That way we drop any outliers (if there are any - mine is) providing more reliable results.

Is it on port 80/8080, or any other port which will be proxied by a transparent (or even non-transparent) proxy? That would provide a real ability for ISPs to skew results.

---
http://blog.jason.pollock.ca

michaeln

220 posts

Master Geek

Trusted

#542692 8-Nov-2011 16:41

jpollock:

Is it on port 80/8080, or any other port which will be proxied by a transparent (or even non-transparent) proxy? That would provide a real ability for ISPs to skew results.

Does that matter? The report is "Time to download a dummy page". I.e., it's explicitly WWW traffic. Anything the ISP does that improves that time is presumably legitimate. After all, that's what most users perceive.

JohnButt

374 posts

Ultimate Geek

Trusted

#542701 8-Nov-2011 16:54

michaeln:
jpollock:

Is it on port 80/8080, or any other port which will be proxied by a transparent (or even non-transparent) proxy? That would provide a real ability for ISPs to skew results.

Does that matter? The report is "Time to download a dummy page". I.e., it's explicitly WWW traffic. Anything the ISP does that improves that time is presumably legitimate. After all, that's what most users perceive.

I agree with @michaeln, tis not our problem if the ISP improves the timing for all users to all pages, in fact that is our desire. We may even publish results to more pages in future to provide greater scope for comparisons.

We have a logical solution for identifying what we determine is intentional gaming by ISPs and are prepared to publish results when we find any attempt.

insane

2415 posts

Uber Geek

Trusted

Subscriber

#542705 8-Nov-2011 17:00

As far as I know some ISPs do prioritise some widely used benchmarks (speedtest.net)

Hopefully TrueNet uses a decent random sample of webpages for testing as it wouldn't take too much effort for an ISP to sniff the DNS requests from a user/staff member with a TrueNet probe and skew results in their favor.

JohnButt

374 posts

Ultimate Geek

Trusted

#542708 8-Nov-2011 17:06

insane: As far as I know some ISPs do prioritise some widely used benchmarks (speedtest.net)

Hopefully TrueNet uses a decent random sample of webpages for testing as it wouldn't take too much effort for an ISP to sniff the DNS requests from a user/staff member with a TrueNet probe and skew results in their favor.

That is exactly what our solution will detect.

jpollock

600 posts

Ultimate Geek

Trusted

#542727 8-Nov-2011 17:49

JohnButt:
michaeln:
jpollock:

Is it on port 80/8080, or any other port which will be proxied by a transparent (or even non-transparent) proxy? That would provide a real ability for ISPs to skew results.

Does that matter? The report is "Time to download a dummy page". I.e., it's explicitly WWW traffic. Anything the ISP does that improves that time is presumably legitimate. After all, that's what most users perceive.

I agree with @michaeln, tis not our problem if the ISP improves the timing for all users to all pages, in fact that is our desire. We may even publish results to more pages in future to provide greater scope for comparisons.

We have a logical solution for identifying what we determine is intentional gaming by ISPs and are prepared to publish results when we find any attempt.

You see, depending on the traffic/probes/etc, it can easily be detected and optimized for. For example, by increasing the cache hold times for the pages which are used, to the point of ignoring the validity period. While this "improves" test results because the page is served from the local cache, it results in the wrong page being returned to the customer.

So, without publication of the test methodology, I can only say it's measuring the performance of the http cache, and easily fudged by ISPs, much the same way that video card manufacturers would detect a performance test and run special code paths which produced visibly buggy results faster.

It is the responsibility of the test designer to design a benchmark that is representative and validate that the results aren't actively subverted by ISPs looking to improve their results. It's good to hear that you've got a solution to identify intentional gaming. Care to share the test methodology, or do I have to sign up and snoop the traffic?

edit: Even without funny business, if all of the terminals are hitting the same pages, they will all hit the cache. The first one will be slow, the rest quick, with an average that doesn't represent random surfing.

---
http://blog.jason.pollock.ca

DonGould

3892 posts

Uber Geek

#542765 8-Nov-2011 19:16

jpollock: Care to share the test methodology, or do I have to sign up and snoop the traffic?

edit: Even without funny business, if all of the terminals are hitting the same pages, they will all hit the cache. The first one will be slow, the rest quick, with an average that doesn't represent random surfing.

How is the information actually used?

I was driving down the road this afternoon and saw a massive bill board claiming that Telstra are the fastest.

Are you suggesting that Telstra are the fastest just because they fudge the figures and Vodafone have been in the back water because they just don't bother?

Is this information being used by consumers to make provider choices? Or is it mainly being used by industry guys to help build business cases to get better get in to the network or the likes of Telecom to sell more capacity to slower providers?

If it's consumers, then yes, I can see a point for not having the figures fudged... but who would fudge it anyway that we wouldn't eventually notice and start posting on GZ naming names? Just look at the long thread about the Telstra link to the US last month as an example.

Do most consumers talk to a trusted tech guy first, and do most of us not see though the rubbish?

If it's ISPs just cheating the stats for industry, then who are they kidding?! ...and what for?

Promote New Zealand - Get yourself a .kiwi.nz domain name!!!

Check out mine - i.am.a.can.do.kiwi.nz - don@i.am.a.can.do.kiwi.nz

JohnButt

374 posts

Ultimate Geek

Trusted

#542773 8-Nov-2011 19:35

jpollock:
It is the responsibility of the test designer to design a benchmark that is representative and validate that the results aren't actively subverted by ISPs looking to improve their results. It's good to hear that you've got a solution to identify intentional gaming. Care to share the test methodology, or do I have to sign up and snoop the traffic?

Yup we benchmark each ISP - snooping will not assist.

1 | 2 | 3