Broadband Performance measurements for May

Forums › New Zealand Broadband › Broadband Performance measurements for May

JohnButt

374 posts

Ultimate Geek

Trusted

#104075 14-Jun-2012 12:10

We have published our measurements for May here

https://www.truenet.co.nz/articles/may-2012-broadband-report

Some ISPs are struggling to keep up, and we have published Maxnet for the first time - although the sample is small compared to other ISPs.

We are still looking for volunteers, needing about 50-100 to fill in gaps.

1 | 2 | 3 | 4 | 5

3711 posts

Uber Geek

Trusted

#640647 14-Jun-2012 12:38

Thanks John, always interesting to read; keep up the good work!

IMO, it would be good to get some more analysis of international performance though. and perhaps some indication of average latency per ISP?

"I was born not knowing and have had only a little time to change that here and there." | Octopus Energy | Sharesies
- Richard Feynman

JohnButt

374 posts

Ultimate Geek

Trusted

#640657 14-Jun-2012 12:45

I plan to publish more info in the next couple weeks. although a little more focused on uploads is my current effort.
Latency per ISP is possible, do you think there would be that much interest? Should I just publish on Geekzone?

SamF

1578 posts

Uber Geek

Trusted

#640671 14-Jun-2012 13:01

+1 for international speeds and latency (both national and international). TBH, national speeds are a bit moot since I'd say for most people the majority of their traffic is from overseas. It's not unusual to see someone with very good local speeds but completely rubbish international speeds.

itxtme

2102 posts

Uber Geek

#640672 14-Jun-2012 13:01

sidefx: Thanks John, always interesting to read; keep up the good work!

IMO, it would be good to get some more analysis of international performance though. and perhaps some indication of average latency per ISP?

Totally agree. I see TC cable always scaling very high but when the international bandwidth is affected (by proxy farm or through mass use) it can be an absolute dog of an ISP

Talkiet

4792 posts

Uber Geek

Trusted

#640675 14-Jun-2012 13:04

John, I'm happy to take this down if you wish, but I quickly added a new scale to your graph... It has not changed any of the data and is annotated as being a modification plus includes a link to the original...

http://en.wikipedia.org/wiki/Wikipedia:Don%27t_draw_misleading_graphs

Cheers - N

Please note all comments are from my own brain and don't necessarily represent the position or opinions of my employer, previous employers, colleagues, friends or pets.

JohnButt

374 posts

Ultimate Geek

Trusted

#640699 14-Jun-2012 13:28

Talkiet,

I agree with the point about misleading graphs, but I'm not sure about the perception of performance. the suppressed zero may be more accurate. (Love your chart :-)

If the speed got to say 20% of the maximum the impact would be more than your graph would hint.
eg, anecdotally: on my home connection I did notice the evening drop to 78% last November, it had severe impact on almost everything I did. With speeds now above 96% of my 15Mb/s I never notice any restrictions on performance.

ie 96% = hard to notice
78% easy to notice, even on a 15Mb/s service

Remember these are averages over a very large number of tests, we do over 1 million a month, or say 100,000 per ISP (guestimate) which means a single slow test, while noticable, is unlikely to impact the results, but many dropped speeds will show in the chart with a suppressed zero.

LennonNZ

2459 posts

Uber Geek

ID Verified

Trusted

#640703 14-Jun-2012 13:34

Can you explain what "advertised speed" means?

Is this the physical connection speed of the Broadband Connection?

For example:

If someone connects at 8M/1M on ADSL1 then the advertised speed is 8M?
If someone connects at 24M/1M on ADSL2 then the advertised speed is 24M?
If someone connects at 50M/10M on VDSL then the advertised speed is 50M?

Or what?

I don't think it will be this as TelstraClear have speeds over 100% which is impossible if you use the above standard?

Equinox IT

Cloud spending continues to surge globally, but most organisations haven’t made the changes necessary to maximise the value and cost-efficiency benefits of their cloud investments. Download the whitepaper From Overspend to Advantage now.

JohnButt

374 posts

Ultimate Geek

Trusted

#640715 14-Jun-2012 13:48

LennonNZ: Can you explain what "advertised speed" means?

Is this the physical connection speed of the Broadband Connection?

For example:

If someone connects at 8M/1M on ADSL1 then the advertised speed is 8M?
If someone connects at 24M/1M on ADSL2 then the advertised speed is 24M?
If someone connects at 50M/10M on VDSL then the advertised speed is 50M?

Or what?

I don't think it will be this as TelstraClear have speeds over 100% which is impossible if you use the above standard?

I covered this last month on our website but I guess it's going to be a regular issue so will include the explanation next time.

Nearly all ISPs advertise speed as "Up to" or "As fast as your line will allow" or "Maximum". TCL cable advertise as 15Mb/s for the probes we are reporting at present.

The report compares each DSL probe with the maximum speed each probe achieves, so 95% of maximum advertised speed is effectively reporting 95% of the DSL speed achieved, usually the peak speed is reached during off-peak hours.

TCL cable - or any Fibre operator, advertise a specific speed, so we compare to that. We are still building fibre probe numbers so are not able to report that yet.

NonprayingMantis

6434 posts

Uber Geek

#640724 14-Jun-2012 13:59

JohnButt:
LennonNZ: Can you explain what "advertised speed" means?

Is this the physical connection speed of the Broadband Connection?

For example:

If someone connects at 8M/1M on ADSL1 then the advertised speed is 8M?
If someone connects at 24M/1M on ADSL2 then the advertised speed is 24M?
If someone connects at 50M/10M on VDSL then the advertised speed is 50M?

Or what?

I don't think it will be this as TelstraClear have speeds over 100% which is impossible if you use the above standard?

I covered this last month on our website but I guess it's going to be a regular issue so will include the explanation next time.

Nearly all ISPs advertise speed as "Up to" or "As fast as your line will allow" or "Maximum". TCL cable advertise as 15Mb/s for the probes we are reporting at present.

The report compares each DSL probe with the maximum speed each probe achieves, so 95% of maximum advertised speed is effectively reporting 95% of the DSL speed achieved, usually the peak speed is reached during off-peak hours.

TCL cable - or any Fibre operator, advertise a specific speed, so we compare to that. We are still building fibre probe numbers so are not able to report that yet.

in other words, it gives telstraclear a huge advantage by using a different methodology for them compared to everyone else.

By measuring DSL as a percentage of the maximum actual, then (obviously) no DSL provider can EVER get higher than 100% (since if their actual speed improves, that just shifts the 100%).

Whereas with Telstraclear, all they have to do is provision a connection a bit faster than the advertised speed, and Bingo: greater than 100% peak speeds!!!

since DSL doesn't have 'advertised speeds' then you obviously can't use that methodlogy for DSL, but there is nothing sotpping you using the same "% of peak actual" method for Cable to normalise the results.

mercutio

1392 posts

Uber Geek

#640768 14-Jun-2012 14:47

NonprayingMantis:
JohnButt:
LennonNZ: Can you explain what "advertised speed" means?

Is this the physical connection speed of the Broadband Connection?

For example:

If someone connects at 8M/1M on ADSL1 then the advertised speed is 8M?
If someone connects at 24M/1M on ADSL2 then the advertised speed is 24M?
If someone connects at 50M/10M on VDSL then the advertised speed is 50M?

Or what?

I don't think it will be this as TelstraClear have speeds over 100% which is impossible if you use the above standard?

I covered this last month on our website but I guess it's going to be a regular issue so will include the explanation next time.

Nearly all ISPs advertise speed as "Up to" or "As fast as your line will allow" or "Maximum". TCL cable advertise as 15Mb/s for the probes we are reporting at present.

The report compares each DSL probe with the maximum speed each probe achieves, so 95% of maximum advertised speed is effectively reporting 95% of the DSL speed achieved, usually the peak speed is reached during off-peak hours.

TCL cable - or any Fibre operator, advertise a specific speed, so we compare to that. We are still building fibre probe numbers so are not able to report that yet.

in other words, it gives telstraclear a huge advantage by using a different methodology for them compared to everyone else.

By measuring DSL as a percentage of the maximum actual, then (obviously) no DSL provider can EVER get higher than 100% (since if their actual speed improves, that just shifts the 100%).

Whereas with Telstraclear, all they have to do is provision a connection a bit faster than the advertised speed, and Bingo: greater than 100% peak speeds!!!

since DSL doesn't have 'advertised speeds' then you obviously can't use that methodlogy for DSL, but there is nothing sotpping you using the same "% of peak actual" method for Cable to normalise the results.

Isn't ADSL up to 8 megabit? So if people are getting 16 megabit then they should show as 200%? I don't remember anyone saying anything about speeds on adsl when the switch to adsl 2 started.. as it's so variable for speeds...

and due to chorus aiming for a snr of 12 it's impossible to get 24megabit adsl in nz.

mercutio

1392 posts

Uber Geek

#640770 14-Jun-2012 14:49

As well as giving mean performance could you give median performance?

mercutio

1392 posts

Uber Geek

#640773 14-Jun-2012 14:53

JohnButt: Talkiet,

I agree with the point about misleading graphs, but I'm not sure about the perception of performance. the suppressed zero may be more accurate. (Love your chart :-)

If the speed got to say 20% of the maximum the impact would be more than your graph would hint.
eg, anecdotally: on my home connection I did notice the evening drop to 78% last November, it had severe impact on almost everything I did. With speeds now above 96% of my 15Mb/s I never notice any restrictions on performance.

ie 96% = hard to notice
78% easy to notice, even on a 15Mb/s service

Remember these are averages over a very large number of tests, we do over 1 million a month, or say 100,000 per ISP (guestimate) which means a single slow test, while noticable, is unlikely to impact the results, but many dropped speeds will show in the chart with a suppressed zero.

I think that depends a lot on the ISP - if there is packet loss nationally then international performance can drop a lot - but if there is extra delay then it's often not the case to the same degree.

Also if there are many dropped speeds then that's at least worth mentioning? Like to me "not being able to use the internet at all" is worse my connection going half speed.

Also considering this is for single tcp connections - and usually in the real world multiple connections are made in parallel - then even with 78% performance there may be 96% performance nationally.

Also sometimes sync rates do vary over time which has nothing to do with the ISP.

LennonNZ

2459 posts

Uber Geek

ID Verified

Trusted

#640774 14-Jun-2012 14:55

If TCL "advertise" 15M and customers are getting more.. then shouldn't this be a fault?
From your results it seems TelstraClear "Advertise" 15M and customers get 19M connection maybe?

If you treat the TelstraClear Cable Connections are 19M then the graph will look just the same as their ADSL connections (Bad at peak)

Wuy don't you ignore the "advertise speed" (As this only applies to the Cable Connections) and use the "max connection of any our our probes on the connection" as use this.

Also.. If an ISP is slow (say their customers only get 10M at peak) and their physical connection speed is 15M then it would show 100% at 10M.

mercutio

1392 posts

Uber Geek

#640775 14-Jun-2012 14:56

With two isp's dropping below 80% - I suppose that's considered a fail. It'd be interesting to know how many of the connections on those two isp's failed. I think in both cases there was handover congestion.

mercutio

1392 posts

Uber Geek

#640777 14-Jun-2012 14:58

LennonNZ: If TCL "advertise" 15M and customers are getting more.. then shouldn't this be a fault?
From your results it seems TelstraClear "Advertise" 15M and customers get 19M connection maybe?

If you treat the TelstraClear Cable Connections are 19M then the graph will look just the same as their ADSL connections (Bad at peak)

Wuy don't you ignore the "advertise speed" (As this only applies to the Cable Connections) and use the "max connection of any our our probes on the connection" as use this.

Also.. If an ISP is slow (say their customers only get 10M at peak) and their physical connection speed is 15M then it would show 100% at 10M.

If an ISP was to shape all of their individual tcp connections arbitarily at a maximum of 10 megabit they may get better performance on these tests - while still giving overall worse performance.

I completely agree TelstraClear is measured badly - I brought it up another time and nothing changed, so I don't think anything will change this time either. Especially considering that Telstraclear are using Truenet in their advertising.

1 | 2 | 3 | 4 | 5