Broadband Performance measurements for May

Forums › New Zealand Broadband › Broadband Performance measurements for May

1 | 2 | 3 | 4 | 5

90 posts

Master Geek

#641051 14-Jun-2012 22:58

JohnButt, great to see this data but I have questions too (pesky geeks!). Why does the 'Speed by time of day' graph not go lower than 80% ? Enquiring minds want to know just how woeful TelstraClear ADSL is between 5 and 11pm.

How do I reconcile the great consistency of Snap with their slow page loads ? I expect people will say "it's the transparent proxies on the big ISPs", so does the Webpage Download test deliberately cache bust when it makes the request ? Sure, normal users don't do that but it would be great to do two tests, and compare apparent and raw pageload times. How does this all fit with the tests described in https://www.truenet.co.nz/how-does-it-work

insane

3239 posts

Uber Geek

ID Verified

Trusted

#641079 15-Jun-2012 00:51

nickt: JohnButt, great to see this data but I have questions too (pesky geeks!). Why does the 'Speed by time of day' graph not go lower than 80% ? Enquiring minds want to know just how woeful TelstraClear ADSL is between 5 and 11pm.

How do I reconcile the great consistency of Snap with their slow page loads ? I expect people will say "it's the transparent proxies on the big ISPs", so does the Webpage Download test deliberately cache bust when it makes the request ? Sure, normal users don't do that but it would be great to do two tests, and compare apparent and raw pageload times. How does this all fit with the tests described in https://www.truenet.co.nz/how-does-it-work

I guess that is only fair to see how low TC DSL goes if their Cable is allowed to go above 100%.

I guess the reason for consistent speeds and slow page load times could be interpreted into that they are simply consistently slow.. however there is more than likely something else at play, such as maybe DNS lookup speed affecting the page loading results.

Ragnor

8219 posts

Uber Geek

Trusted

#641080 15-Jun-2012 00:54

+1 for changing the cable results from advertised/marketing speed to measured maximum speed like the rest of the results.

sidefx

3711 posts

Uber Geek

Trusted

#641163 15-Jun-2012 09:52

Ragnor: +1 for changing the cable results from advertised/marketing speed to measured maximum speed like the rest of the results.

Then to hold their position at the "top" they could just artificially limit all customers to their advertised speeds and hey presto they have 100% all the time... Though admittedly it wouldn't be as over inflated as the current "lead" but still. Maybe given the differences between fibre and adsl it would be better to put it on a seperate graph. Or just keep it in mind when looking at the graphs. Must admit it doesn't bother me as much as some, lol

"I was born not knowing and have had only a little time to change that here and there." | Octopus Energy | Sharesies
- Richard Feynman

NonprayingMantis

6434 posts

Uber Geek

#641211 15-Jun-2012 10:48

sidefx:
Ragnor: +1 for changing the cable results from advertised/marketing speed to measured maximum speed like the rest of the results.

Then to hold their position at the "top" they could just artificially limit all customers to their advertised speeds and hey presto they have 100% all the time...

and that would be absolutely fine since then it would a genuine comparison using the same methodology

sidefx

3711 posts

Uber Geek

Trusted

#641225 15-Jun-2012 11:01

NonprayingMantis:

and that would be absolutely fine since then it would a genuine comparison

See IMHO that would just skew the results in the opposite direction; given that ADSL is usually advertised as "Maximum speed" (whatever your line supports) whereas cable is advertised as "Up to 15/2" and the graph is an attempt to show if you're "getting what you pay for" I think in some ways it's fair enough as is... though one could argue it's also comparing apples and oranges (analogue v digital) and perhaps the cable\fibre providers should be on a seperate graph.

TBH, just an explanation as John has proposed would make just as much sense...

"I was born not knowing and have had only a little time to change that here and there." | Octopus Energy | Sharesies
- Richard Feynman

NonprayingMantis

6434 posts

Uber Geek

#641244 15-Jun-2012 11:26

sidefx:
NonprayingMantis:

and that would be absolutely fine since then it would a genuine comparison

See IMHO that would just skew the results in the opposite direction; given that ADSL is usually advertised as "Maximum speed" (whatever your line supports) whereas cable is advertised as "Up to 15/2" and the graph is an attempt to show if you're "getting what you pay for" I think in some ways it's fair enough as is... though one could argue it's also comparing apples and oranges (analogue v digital) and perhaps the cable\fibre providers should be on a seperate graph.

TBH, just an explanation as John has proposed would make just as much sense...

it would skew the results back towards a genuine comparison - whihc is what we want

If TCL decided to set people’s max speed at the speed they pay for (15) and they achieved that 100% of the time, then good for them. the chart would show them achieving their speeds 100% and they would justifiable have the top score of all ISPs

As it stands now, the scores are not comparable since, it is IMPOSSIBLE for any DSL performer to get above 100%. This is because of the decision to use a different methodology for Cable than for DSL and not anything to do with real world actual performance.

Consider a pair bonded VDSL line in idealised settings chieving actual speeds of, say, 100Mbps 100% of the time. Even under that extreme scenario, the VDSL would still look WORSE than the cable since Truenet chooses to use a different comparison for DSL which restricts DSL from ever being able to get above 100%

GoodSync

GoodSync. Easily back up and sync your files with GoodSync. Simple and secure file backup and synchronisation software will ensure that your files are never lost (affiliate link).

JohnButt

374 posts

Ultimate Geek

Trusted

#641414 15-Jun-2012 14:52

Great discussion, the major point I take from this is that people do understand what the different lines mean and why they are where they are, which is after-all what we are most interested in achieving. Many clearly disagree with the premise that Cable should be shown with 120% of advertised speed while the best ADSL can do is 100% of advertised speed when the ads say "Maximum" or similar, but I see some agree with me that this is the best way to show it.

However, I am keen to find a great way (different from my current charts) to show throughput performance in a fair and equitable way for ADSL/VDSL. To this end I am attaching an interesting chart for discussion that demonstrates a number of things, but especially that throughput is very dependent on distance. A number of posts show that this is not well understood.
The spots represent single probe throughput (down) at 5am averaged over every day this month, the speed is the vertical column and distance the horizontal column.
The curves are standard excel log curves (not always helpful) with notes:
1 the relative performance by ISP for the probes we have connected to them - higher may be because they get their customers a better deal, but is also potentially to be due to random distribution (The sample is 20-50 per ISP)
2 The length of the line shows the distribution of each ISP, eg Telecom & Vodafone have probes a long way out, Snap does not have one very close in or far out.
3 Caution, TelstraClear DSL & Xnet sample may be a little too small in comparison - we need more volunteers for these two
4 One VDSL probe is off-scale - needs investigating, these are not in our monthly report.

It appears to me from the distribution of speeds for close in probes that there are many probes on lines with poor local network connections, or maybe poor modems - more research is needed. These are not ISP issues necessarily.

Comments welcome.

Distances are approximate, they are calculated by measuring the distance from the address to the nearest DSLAM in a cabinet or exchange, errors could include location of the Cabinet or indeed a Cabinet may have been installed since the measurement

Talkiet

4792 posts

Uber Geek

Trusted

#641428 15-Jun-2012 15:07

Gah - no good - I tried to stay out as long as possible and not actually give an opinion but I can't do it :-) (MY OPINION BTW)

I think comparing Cable with a declared maximum performance to ADSL with derived (and presumably adaptive) maximum performance is misleading.

As has been pointed out, an ADSL provider could game the numbers by stating "2mbps is the max advertised speed" however for many political reasons no-one will do that.

On the other hand, what are you going to do when you start getting Cable customers on the warpspeed plans with the description of "WarpSpeed 100G Up to 100 Mbps / 10 Mbps 100G $115.95 2G packs at $2.95 per pack" - an in fact how are you sure that some of the probes you already have aren't on lines that have since been upgraded to 100mbps?

Presumably 100% will now be set at 100Mbps which clearly the users are unlikely to attain.

At that point, you need to make a decision to either change your Cable baseline to adopt a similar method as ADSL for determining "100%" or start plotting some hideously bad figures for TCL cable.

Please indulge us and state what approach you will take, or are you going to not accept any 100mbps cable customers because it will dramatically expose the inequity of measurement criteria? (as opposed to the 15mbit customers which only partly expose it! :-)

Cheers - N

Please note all comments are from my own brain and don't necessarily represent the position or opinions of my employer, previous employers, colleagues, friends or pets.

JohnButt

374 posts

Ultimate Geek

Trusted

#641441 15-Jun-2012 15:23

Thanks Talkiet, you have just suggested a very good solution. We are gathering 100Mb/s Cable from existing probes (we recognize them quickly :-) and have accepted a few new, however we are having to work on a new probe for them so will not be ready for a while, however we also have some fibre, and these have speeds all over the place, depending on the chosen speed mostly. Hence a separate chart makes a lot of sense fro technologies where the speed is selected.

Talkiet

4792 posts

Uber Geek

Trusted

#641446 15-Jun-2012 15:30

JohnButt: Thanks Talkiet, you have just suggested a very good solution. We are gathering 100Mb/s Cable from existing probes (we recognize them quickly :-) and have accepted a few new, however we are having to work on a new probe for them so will not be ready for a while, however we also have some fibre, and these have speeds all over the place, depending on the chosen speed mostly. Hence a separate chart makes a lot of sense fro technologies where the speed is selected.

Cool, does that mean you're coming around to the opinion that including the static 15mbps based figures from existing Cable customers on the existing graph may not be a completely valid data presentation? :-) If you accept the 100mbps (or fibre) data points have no place on the graph then I can't see how you could support having the 15mbps cable data there.

The real issue here (and you hint at it) is that as access speeds increase, there isn't a linear peak hour degradation for many many reasons.

I like the comparison across ISPs of the performance by time of day, and even with the constraints (limited number of sites, one protocol only etc, lack of total control of user activity, effects of interleaving settings etc) it's interesting data - please don't stop it.

Cheers - N

Please note all comments are from my own brain and don't necessarily represent the position or opinions of my employer, previous employers, colleagues, friends or pets.

Shindig

1585 posts

Uber Geek

Trusted

#641515 15-Jun-2012 17:20

Thank you for this information. It is really interesting to read.

The little things make the biggest difference.

insane

3239 posts

Uber Geek

ID Verified

Trusted

#641551 15-Jun-2012 18:16

This is why people have been saying that you cannot look at speed and consistency separately, they need to be looked at together, and the 100% point for any given probe needs to be the max speed that probe has ever achieved, not the line speed.

matts231

29 posts

Geek

#641697 16-Jun-2012 00:18

LennonNZ: If TCL "advertise" 15M and customers are getting more.. then shouldn't this be a fault?

Telstra... on their cable networks either side of the tasman appear to over provision HFC services by around 20%. Who knows why, but it's a nice bonus.

JohnButt

374 posts

Ultimate Geek

Trusted

#641825 16-Jun-2012 15:14

insane: This is why people have been saying that you cannot look at speed and consistency separately, they need to be looked at together, and the 100% point for any given probe needs to be the max speed that probe has ever achieved, not the line speed.

Exactly how we do this comparison.

1 | 2 | 3 | 4 | 5