Broadband Performance measurements for May

Forums › New Zealand Broadband › Broadband Performance measurements for May

1 | 2 | 3 | 4 | 5

1392 posts

Uber Geek

#640780 14-Jun-2012 15:02

a more honest approach at real performance may be to look at interpacket delay and packet loss. When there is packet loss or significant delay in packets this is often when performance starts to bottle neck, and congestion is experienced.

if you look at minimum and maximum interpacket delay you should be able to guage how well a connection is operating even when operating below the maximum "advertised speed".

ArcticSilver

729 posts

Ultimate Geek

#640824 14-Jun-2012 16:12

First of all, thanks for the results! It's nice to see some one actually benchmarking our ISP's.

Secondly, I do agree with the others, the graphics are a little misleading.

What I personally would like to see would be:

- ISP's average speed, in Mbps (not percentages). Possibly with a note where the "max" is lower than the standard ADSL2.
- ISP's average latency to Australia and the US.
- ISP's average speed (as above) to Australia and the US (the later being most important).

The later of the 3 I believe is the most important. It is very easy for an ISP to perform very well locally, but then drop the ball internatially and this is the area most people will notice. It should also avoid using websites such as youtube for the bechmark (possibly even have a seperate youtube benchmark) as a lot of ISP's here cache youtube, which may skew the results in an ISP's favor.

Latency is also a big factor, if you are following some strange route to US sites you'll notice a delay in loading a page, it just wont feel as "snappy". It is also a factor for those of us who game, where latency can be the deciding factor.

Edit: As above, packet loss is also a big factor, if any ISP is experiencing it, this will reflect poorly on their performance.

SamF

1578 posts

Uber Geek

Trusted

#640834 14-Jun-2012 16:23

ArcticSilver:What I personally would like to see would be:

- ISP's average speed, in Mbps (not percentages). Possibly with a note where the "max" is lower than the standard ADSL2.

The problem with this is that download speeds are limited by ADSL sync speeds, which are for the most part out of the ISP's control. This is why the stats are currently relative to ADSL sync speeds as it's the only way to eliminate this part of the equation.

JohnButt

374 posts

Ultimate Geek

Trusted

#640839 14-Jun-2012 16:29

mercutio:
If an ISP was to shape all of their individual tcp connections arbitarily at a maximum of 10 megabit they may get better performance on these tests - while still giving overall worse performance.

I completely agree TelstraClear is measured badly - I brought it up another time and nothing changed, so I don't think anything will change this time either. Especially considering that Telstraclear are using Truenet in their advertising.

I object to your last phrase. TrueNet is an independent measuring company with absolutely no relationship with TelstraClear, we work very hard to ensure that all ISPs are treated equally and we are very open about our criteria, hence my involvement on this forum.

I have not changed the presentation because I do not agree with you.

Simply put, if we publish achievement related to advertised speed, where the performance falls below advertised speed, then we are identifying that an improvement is necessary, we believe this is true for any variation below 95%.

Publishing an achievement that has been improved to where the minimum is above the advertised speed seems like a win-win to me. We published results earlier compared to 15Mb/s when TelstraClear were only delivering 15Mb/s max, with a drop to less than 12Mb/s during peak hours, they fixed that by increasing their speeds at all times - I cannot see a problem with this, if I were a TelstraClear Cable customer I would be very happy with TrueNet's presentation and my extra 3-4Mb/s.

mercutio

1392 posts

Uber Geek

#640856 14-Jun-2012 17:02

JohnButt:
mercutio:
If an ISP was to shape all of their individual tcp connections arbitarily at a maximum of 10 megabit they may get better performance on these tests - while still giving overall worse performance.

I completely agree TelstraClear is measured badly - I brought it up another time and nothing changed, so I don't think anything will change this time either. Especially considering that Telstraclear are using Truenet in their advertising.

I object to your last phrase. TrueNet is an independent measuring company with absolutely no relationship with TelstraClear, we work very hard to ensure that all ISPs are treated equally and we are very open about our criteria, hence my involvement on this forum.

I have not changed the presentation because I do not agree with you.

Simply put, if we publish achievement related to advertised speed, where the performance falls below advertised speed, then we are identifying that an improvement is necessary, we believe this is true for any variation below 95%.

Publishing an achievement that has been improved to where the minimum is above the advertised speed seems like a win-win to me. We published results earlier compared to 15Mb/s when TelstraClear were only delivering 15Mb/s max, with a drop to less than 12Mb/s during peak hours, they fixed that by increasing their speeds at all times - I cannot see a problem with this, if I were a TelstraClear Cable customer I would be very happy with TrueNet's presentation and my extra 3-4Mb/s.

There's still an inherent bias whether or not a relationship exists.

What about local loop unbundling when other ISP's can set SNR margins below what Chorus set - and achieve higher sync rates - or how Xtra by having interleaving on default are decreasing both sync rates and maximum performance.

So if ADSL providers want to be measured as over 100% they should advertise speeds of up to 10 megabit? Then if some users get 15 megabit or higher then they can be ranked as 150%?

Maybe I'm mistaken, but I've never seen any ISP in New Zealand advertising up to 24 megabit ADSL. And doing a Google search I only come across Telstraclear: http://www.telstraclear.co.nz/residential/homeplan/internet/broadband/speed-and-usage.cfm

When I have used TelstraClear cable in the past, I have found performance to be sluggish - worse than DSL - poor international speeds unless don't multiple connections at once, and multiple periods of degredation. That said they have reasonably often commented about "increasing their network capacity".

International speeds wise Telstraclear cable can often be around 1 to 2 megabit. So if you compare that to 18 or 19 megabit they're at a fraction of their total speed.

On top of that - I've experienced multiple times when pages just won't load, on multiple different connections with multiple different computers, not just this year, but in previous years when other activity works.

mercutio

1392 posts

Uber Geek

#640859 14-Jun-2012 17:07

SamF:
ArcticSilver:What I personally would like to see would be:

- ISP's average speed, in Mbps (not percentages). Possibly with a note where the "max" is lower than the standard ADSL2.

The problem with this is that download speeds are limited by ADSL sync speeds, which are for the most part out of the ISP's control. This is why the stats are currently relative to ADSL sync speeds as it's the only way to eliminate this part of the equation.

With local loop unbundling there can be both positive and negative speed differences, neither of which will be picked up by this system.

If you are cabinetised and still on LLU, then you will show as more often reaching your maximum speed, even though you have subpar performance.

If you are stuck on LLU near an exchange, and the ISP sets a lower SNR margin, you can easily get higher sync rates, and higher performance, but your "maximum" will increase, and everything is compared to that.

LennonNZ

2459 posts

Uber Geek

ID Verified

Trusted

#640861 14-Jun-2012 17:12

The problem I see if you are mixing Absolute Average Test with Relative Average tests on the same graph.

Telstra Clear is Absolute average speed (with 100% being 15M)
Everyone else you are Getting the Maximum Speed any test did and then thats 100%

Mixiing these on the same graph doesn't make sense at all IMHO.

The question is Why don't you treat TelstraClear the same as everyone else in the way you represent them (i.e. see the maximum speed someone got and then have that as 100%)

Quic Broadband

Move to New Zealand's best fibre broadband service (affiliate link). Free setup code: R587125ERQ6VE. Note that to use Quic Broadband you must be comfortable with configuring your own router.

mercutio

1392 posts

Uber Geek

#640865 14-Jun-2012 17:18

ArcticSilver: First of all, thanks for the results! It's nice to see some one actually benchmarking our ISP's.

Secondly, I do agree with the others, the graphics are a little misleading.

What I personally would like to see would be:

- ISP's average speed, in Mbps (not percentages). Possibly with a note where the "max" is lower than the standard ADSL2.
- ISP's average latency to Australia and the US.
- ISP's average speed (as above) to Australia and the US (the later being most important).

The later of the 3 I believe is the most important. It is very easy for an ISP to perform very well locally, but then drop the ball internatially and this is the area most people will notice. It should also avoid using websites such as youtube for the bechmark (possibly even have a seperate youtube benchmark) as a lot of ISP's here cache youtube, which may skew the results in an ISP's favor.

It can happen the other way around as well.

With youtube, people often think the caching is the problem for some reason. It's often not the cached videos that struggle - it's the non-cached. And if you try the same video multiple times the server can change. If you look at resource monitor of a video loading slowly you can find out the tcp address. You can then use something like pingplotter to trace to it.

Some of the slow loading videos can be seen to be coming from europe etc.

Have you looked at smokeping before? It can show packet loss and latency with nice graphs.

Latency is also a big factor, if you are following some strange route to US sites you'll notice a delay in loading a page, it just wont feel as "snappy". It is also a factor for those of us who game, where latency can be the deciding factor.

You'd think. I've actually been experimenting with this, and it's not actually as much of an issue as I first imagined.

If you have a fast CPU/disk server in Texas, and a slow server in Los Angeles, you can expect the Texas server to give faster page load times with dynamic pages.

That said it does make a difference - especially if there are lots of links, and/or levels of packet loss, and/or suboptimal server configuration.

Take godaddy as a well known site that has poor performance over higher latency links.

For gaming packet loss can make a bigger difference for things like WoW/Diablo which are TCP/IP based. For UDP traffic a lot of them can take some level of packet loss. There are different kinds of packet loss which don't always show in statistics though. Basically burst packet loss - with lots of dropped packets in a row, but otherwise fine, or low grade frequently.

Edit: As above, packet loss is also a big factor, if any ISP is experiencing it, this will reflect poorly on their performance.

It's not too bad if you have packet loss around 0.1%, but 0.5% is pretty bad. 5% is atrocious.

SamF

1578 posts

Uber Geek

Trusted

#640870 14-Jun-2012 17:21

The point here is that this graph is a rating against Advertised speeds. ADSL connections are always advertised as 'Maximum' ie; the maximum your line can connect at, Telstra advertise their cable service at 15mbits, but for technical reasons, are able to supply faster rates than these. Sure, you could argue (as you are) that Telstra cable connections should be measured at their actual maximum capable speeds, but that would actually unfairly disadvantage them in this graph. At the end of the day, they are supplying faster local speeds than they advertise, and that's a good thing!!

ArcticSilver

729 posts

Ultimate Geek

#640872 14-Jun-2012 17:27

SamF:
ArcticSilver:What I personally would like to see would be:

- ISP's average speed, in Mbps (not percentages). Possibly with a note where the "max" is lower than the standard ADSL2.

The problem with this is that download speeds are limited by ADSL sync speeds, which are for the most part out of the ISP's control. This is why the stats are currently relative to ADSL sync speeds as it's the only way to eliminate this part of the equation.

I fully understand this, but in reality this would be very hard to factor in. I believe statistics, in Mbps on average should show dramatic differences between ISP's, even though it may be slightly skewed because of this.

The only way to realistically fix this problem would be to have Truenet modems which record the line statistics, which may not be economical/practical.

Some better tests may be one's comaring the peek time fluxuation of an ISP's connection and the differences between national and international speeds. This would mean you should still have a national reference point, factoring in the issue of lower sync rates. This is in my mine about the closest you could get to finding out a users sync rate, without being a modem.

NonprayingMantis

6434 posts

Uber Geek

#640877 14-Jun-2012 17:36

SamF: The point here is that this graph is a rating against Advertised speeds. ADSL connections are always advertised as 'Maximum' ie; the maximum your line can connect at, Telstra advertise their cable service at 15mbits, but for technical reasons, are able to supply faster rates than these. Sure, you could argue (as you are) that Telstra cable connections should be measured at their actual maximum capable speeds, but that would actually unfairly disadvantage them in this graph. At the end of the day, they are supplying faster local speeds than they advertise, and that's a good thing!!

whihc is exaclty what makes the comparison entirely silly. Just consider whether iot would be possible for a DSL line to EVER get above 100% on this test even if they were getting speeds of 50Mbps. (It isn't, since then their 100% would be at 50Mbps) therefore the comparison is silly

since the methodology for ADSL is to show comparisons with maximum achieved speed, then the SAME methodology should be used for Cable. That is all people are asking for - a genuine comparison, not a comparison using different methods that gives Cable an unfair advantage.

Maybe to get this to change we need an ISP to launch a product advertised with 2Mbps, but still give the maximum ADSL2+ speeds, then the graph will just start to look silly as that plan will show 1000% over advertised speed!

Talkiet

4793 posts

Uber Geek

Trusted

#640891 14-Jun-2012 18:16

Please, everyone put aside the name of the company I work for, and evaluate this post only on the content. I am fanatically interested in performance and accuracy of reporting. I'm not going after anyone or anything here except the truth and accuracy...

JohnButt: Talkiet,
[snip]
If the speed got to say 20% of the maximum the impact would be more than your graph would hint.
eg, anecdotally: on my home connection I did notice the evening drop to 78% last November, it had severe impact on almost everything I did. With speeds now above 96% of my 15Mb/s I never notice any restrictions on performance.

ie 96% = hard to notice
78% easy to notice, even on a 15Mb/s service
[snip]

JohnButt: (from another post)
Publishing an achievement that has been improved to where the minimum is above the advertised speed seems like a win-win to me. We published results earlier compared to 15Mb/s when TelstraClear were only delivering 15Mb/s max, with a drop to less than 12Mb/s during peak hours, they fixed that by increasing their speeds at all times - I cannot see a problem with this, if I were a TelstraClear Cable customer I would be very happy with TrueNet's presentation and my extra 3-4Mb/s.

John... In your direct reply to me you suggest that a drop of ~20% (to 78%) had a noticable ("severe impact on almost everything I did") effect.

Your other reply seems to suggest however that because the TCL(cable) figures are above an arbitrary value, that the 20% (approx) drop from peak won't have an impact on general use.

Have I interpreted your statements correctly? ie. ADSL performance dropping 20% from max causes bad problems, but that Cable performance dropping 20% from max isn't an issue?

Regards
Neil G

Please note all comments are from my own brain and don't necessarily represent the position or opinions of my employer, previous employers, colleagues, friends or pets.

mercutio

1392 posts

Uber Geek

#640892 14-Jun-2012 18:18

NonprayingMantis:
SamF: The point here is that this graph is a rating against Advertised speeds. ADSL connections are always advertised as 'Maximum' ie; the maximum your line can connect at, Telstra advertise their cable service at 15mbits, but for technical reasons, are able to supply faster rates than these. Sure, you could argue (as you are) that Telstra cable connections should be measured at their actual maximum capable speeds, but that would actually unfairly disadvantage them in this graph. At the end of the day, they are supplying faster local speeds than they advertise, and that's a good thing!!

whihc is exaclty what makes the comparison entirely silly. Just consider whether iot would be possible for a DSL line to EVER get above 100% on this test even if they were getting speeds of 50Mbps. (It isn't, since then their 100% would be at 50Mbps) therefore the comparison is silly

since the methodology for ADSL is to show comparisons with maximum achieved speed, then the SAME methodology should be used for Cable. That is all people are asking for - a genuine comparison, not a comparison using different methods that gives Cable an unfair advantage.

Maybe to get this to change we need an ISP to launch a product advertised with 2Mbps, but still give the maximum ADSL2+ speeds, then the graph will just start to look silly as that plan will show 1000% over advertised speed!

well hangon don't adsl circuits have a cir of 48kbit?

Talkiet

4793 posts

Uber Geek

Trusted

#640893 14-Jun-2012 18:19

mercutio: [snip]

well hangon don't adsl circuits have a cir of 48kbit?

No

Cheers -N

Please note all comments are from my own brain and don't necessarily represent the position or opinions of my employer, previous employers, colleagues, friends or pets.

mercutio

1392 posts

Uber Geek

#640897 14-Jun-2012 18:26

Talkiet: Please, everyone put aside the name of the company I work for, and evaluate this post only on the content. I am fanatically interested in performance and accuracy of reporting. I'm not going after anyone or anything here except the truth and accuracy...

JohnButt: Talkiet,
[snip]
If the speed got to say 20% of the maximum the impact would be more than your graph would hint.
eg, anecdotally: on my home connection I did notice the evening drop to 78% last November, it had severe impact on almost everything I did. With speeds now above 96% of my 15Mb/s I never notice any restrictions on performance.

ie 96% = hard to notice
78% easy to notice, even on a 15Mb/s service
[snip]

JohnButt: (from another post)
Publishing an achievement that has been improved to where the minimum is above the advertised speed seems like a win-win to me. We published results earlier compared to 15Mb/s when TelstraClear were only delivering 15Mb/s max, with a drop to less than 12Mb/s during peak hours, they fixed that by increasing their speeds at all times - I cannot see a problem with this, if I were a TelstraClear Cable customer I would be very happy with TrueNet's presentation and my extra 3-4Mb/s.

John... In your direct reply to me you suggest that a drop of ~20% (to 78%) had a noticable ("severe impact on almost everything I did") effect.

This in itself completely depends on why speeds are going at that speed also. I don't have a convenient way to generate packet loss - but I imagine that it'd be something like 3% packet loss, 10 msec increased ping, and 5 to 10 msec jitter to give 80% nz performance. That internationally would lead to speeds more like 5 to 10% of the 80% reduced speed.

But if instead it was 40 to 50 msec increased pings with 0.5% packet loss, the performance impact on international may be more like 20% of the reduced 80% speed.

Your other reply seems to suggest however that because the TCL(cable) figures are above an arbitrary value, that the 20% (approx) drop from peak won't have an impact on general use.

Have I interpreted your statements correctly? ie. ADSL performance dropping 20% from max causes bad problems, but that Cable performance dropping 20% from max isn't an issue?

I think the problem is that a lot of people think that some number like 8 or 12 says how fast their internet connection is going. I started looking to see what speeds various isp's said they could do, and I kept coming across speed tests!

I even find people who aren't strongly technically savvy running speed tests when their internet is performing badly. And then their speed tests are fine but their net has problems.

All they can do is compare their speed tests to other speed tests.

It'd be interesting to do a/b blind testing, on a fast network connection, and generate packet loss, increased latency, stick speed limits on etc, and see what people notice and what they don't. My gut feeling is that people don't really notice the difference between 10 and 20 megabit in casual internet use, but notice the difference between 1 and 2 megabit, and notice the huge timeouts that packet loss can create.

1 | 2 | 3 | 4 | 5