Commerce Commission selects TrueNet to test broadband quality

Forums › New Zealand Broadband › Commerce Commission selects TrueNet to test broadband quality

freitasm

BDFL - Memuneh
79250 posts

Uber Geek

Administrator

ID Verified

Trusted

Geekzone

Lifetime subscriber

#99243 15-Mar-2012 10:00

Just received:

The Commerce Commission announced today it has selected TrueNet to conduct testing to measure the quality of the broadband services being provided in New Zealand.

“The TrueNet testing is in addition to the Epitiro testing we are already doing, and is more oriented towards measuring the kind of performance that is delivered to people’s homes,” said Dr Ross Patterson, Telecommunications Commissioner.

The Commission will publish reports showing the results of the testing. The frequency of these reports is yet to be confirmed.

TrueNet is looking for more volunteers to take part in their broadband testing. For more information and to volunteer, visit their website at www.truenet.co.nz/get-involved-become-volunteer-tester

Background

The testing is conducted under Section 9A of the Telecommunications Act 2001, which empowers the Commission to conduct inquiries, reviews and studies into any matter relating to the telecommunications industry for the long-term benefit of end-users of telecommunications services within New Zealand.

TrueNet is a New Zealand based company that measures and reports on broadband performance in New Zealand. It is a joint venture between the open source information technology company, Catalyst IT, and Jonette Consulting, a company specialising in broadband reporting analysis.

BarTender

3606 posts

Uber Geek

ID Verified

Trusted

Lifetime subscriber

#595495 15-Mar-2012 10:06

Do you know if TrueNet plan to supply a ADSL modem rather than just a Ethernet router so that it can gather the ADSL stats from the line when it's running the test?

Plus also gather data on if customers are connected to a cabinet / exchange. How far they are away from said cabinet / exchange which could be calculated based on their attenuation.

Are they also going to ensure they download off a range of sites nationally and internationally. As pulling the single file off TradeMe is good to test raw performance into TradeMe, but are we sure it's not being cached by a local cache server sitting inside the ISPs network. Or being pulled from a local CDN? Gathering stats showing when you do the TCP SYN to connect, when the connection is ACKed, when the page starts to download and when it finishes are really important stats showing the end-customer experience based on multiple sites.

Another thing worth testing is DNS response times and depending on the reply if this impacts browsing experience or not.

Browsing experience is based a whole lot more than just the ability to download a single page off a single site.

Lastly to then compare against what the end-user is currently getting vs what they could be getting if their internal wiring was up to scratch based off data that Chorus has as part of the PreQual process?

JohnButt

374 posts

Ultimate Geek

Trusted

#595529 15-Mar-2012 11:21

We are very pleased to have the Commerce Commission as our anchor customer.

The TrueNet system relies on measuring customer's internet performance from a probe connected in volunteers homes and small businesses, free of interference from busy computers and other equipment.

TrueNet has published on it's website monthly performance reports for a small range of key measurements since August 2011. In that time we have seen significant improvement in the performance of ISPs for these measurements. With the Commerce Commission now publishing our information, we are confident we can continue to drive improvements in New Zealand.

With the Commissions support, we hope to be able to report a broader range of measurements from our extensive collection of data. Since the start of our testing from volunteer's homes in 2010, TrueNet has completed 11million test runs with 5 to 10 tests per run. With this contract our test probes will increase three-fold, providing an extensive resource of internet statistics and insights.

To become a TrueNet volunteer go to www.truenet.co.nz/get-involved-become-volunteer-tester

Talkiet

4792 posts

Uber Geek

Trusted

#595532 15-Mar-2012 11:27

Congrats John... Glad to see you're going to be able to expand the testing.

I have a bunch of tech questions about what this extra funding may help you to do but I'll try and catch up with you in Wellington over the next week or so...

Cheers - Neil G

Please note all comments are from my own brain and don't necessarily represent the position or opinions of my employer, previous employers, colleagues, friends or pets.

JohnButt

374 posts

Ultimate Geek

Trusted

#595543 15-Mar-2012 11:36

OK, here are some answers

BarTender: Do you know if TrueNet plan to supply a ADSL modem rather than just a Ethernet router so that it can gather the ADSL stats from the line when it's running the test?

Plus also gather data on if customers are connected to a cabinet / exchange. How far they are away from said cabinet / exchange which could be calculated based on their attenuation.

Are they also going to ensure they download off a range of sites nationally and internationally. As pulling the single file off TradeMe is good to test raw performance into TradeMe, but are we sure it's not being cached by a local cache server sitting inside the ISPs network. Or being pulled from a local CDN? Gathering stats showing when you do the TCP SYN to connect, when the connection is ACKed, when the page starts to download and when it finishes are really important stats showing the end-customer experience based on multiple sites.

Another thing worth testing is DNS response times and depending on the reply if this impacts browsing experience or not.

Browsing experience is based a whole lot more than just the ability to download a single page off a single site.

Lastly to then compare against what the end-user is currently getting vs what they could be getting if their internal wiring was up to scratch based off data that Chorus has as part of the PreQual process?

GREAT QUESTIONS

An ADSL modem would prohibit LAN control for a volunteer and increase our costs considerably due to support requirements - we would need lots of new software. Not keen to do this.
We do gather distance to DSLAM and have published results. I expect to publish more, the last time the results were heading quickly towards theoretical with an average speed of 9.1Mb/s due to the introduction of cabinets shortening the copper lines.
More sites and more tests are in our DNA, just needed the opportunity to conduct such tests, expect that soon.
We do test DNS response times, I have a release almost ready to go, just polishing the comments.
We also do tests to a single international site, but here again we are extremely keen to publish once we have more sites - on International, 1 is not a survey
Our tests are "what they could be getting" from the location of their probe. We recommend that the probe is as close as possible to the drop-lead, hence avoiding any odd internal wiring.

Thanks for the opportunity to provide some feedback. We are keen for new ideas, so I will be taking some of your paragraph 3 comments on-board to see what we can do.

Ragnor

8218 posts

Uber Geek

Trusted

#595895 15-Mar-2012 23:12

Congratulations Jon, well done on getting TrueNet selected!

insane

3236 posts

Uber Geek

ID Verified

Trusted

#595910 15-Mar-2012 23:55

I think this is hopefully a good platform to see results from even the smaller ISPs which don't typically get mentioned unless they have an issues within the testing period.

There are a number of regional or smaller ISPs out there which can and do offer great service, just don't get to hear from them much due to presumably a small sample set for their probes.

One problem I see with the results is with ISPs which have introduced Unlimited plans. Surly users on such plans would bring down the ISPs results, given that their internet traffic is given less priority compared to users on capped plans?

lucky015

743 posts

Ultimate Geek

ID Verified

Trusted

#595915 16-Mar-2012 00:12

insane: I think this is hopefully a good platform to see results from even the smaller ISPs which don't typically get mentioned unless they have an issues within the testing period.

There are a number of regional or smaller ISPs out there which can and do offer great service, just don't get to hear from them much due to presumably a small sample set for their probes.

One problem I see with the results is with ISPs which have introduced Unlimited plans. Surly users on such plans would bring down the ISPs results, given that their internet traffic is given less priority compared to users on capped plans?

I would imagine there would be some way of telling the difference either by Requesting the current plan details upon setting up one of the probes or by seeing which Dynamic IP Address pool the IP is from.

Equinox IT

Cloud spending continues to surge globally, but most organisations haven’t made the changes necessary to maximise the value and cost-efficiency benefits of their cloud investments. Download the whitepaper From Overspend to Advantage now.

sbiddle

30853 posts

Uber Geek

Retired Mod

Trusted

Biddle Corp

Lifetime subscriber

#595940 16-Mar-2012 07:57

Speeds are fine, but they can't show artificial constraints that are affecting the connection. Smaller ISP's could easily have seriously skewed results if they're based off a small number of clients who have poor sync speeds.

I'd love to see somebody do some serious data matching with the raw data from the Chorus ISAM's cross referenced with the customer's address. Chorus know what the estimated dB noise figures and sync speeds are to a property demarc, and from their ISAM also know what the actual sync speed of a current DSL connection is, and whether the line/connection is stable or unstable.

With this data you could very easily identify the large number of homes with internal wiring issues that are affecting their speed, and upsell these people to a master filter install to fix their problems.

I'd also be in favour of Truenet being given Chorus Wireline estimated sync stats for each of their customers, and for Truenet to establish the modem sync stats for client connections along with their real world speeds.

Until people can get it into their heads that internal house wiring is what's broken in NZ, not broadband as a whole, we're not going to make much progress in the DSL world despite the fact it will still remain as the primary delivery platform for most householes in the country for the next ~5 years until the UFB focus moves to residential.

BarTender

3606 posts

Uber Geek

ID Verified

Trusted

Lifetime subscriber

#596079 16-Mar-2012 12:30

sbiddle: Speeds are fine, but they can't show artificial constraints that are affecting the connection. Smaller ISP's could easily have seriously skewed results if they're based off a small number of clients who have poor sync speeds.

...

Funnily enough Steve I sent the exact same email to John about that exact same issue too.

I am not sure if John would be allowed to get access to Wireline and let him gather stats for the customer or how exactly that could work. As you need to be an ISP to gain access to Wireline.

JohnButt

374 posts

Ultimate Geek

Trusted

#596103 16-Mar-2012 13:34

I would love access, plan to approach to ask for it again ASAP :-) Steve's suggested queries would be a great addition to assist us to focus on internal wiring.

mercutio

1392 posts

Uber Geek

#597237 19-Mar-2012 16:06

i strongly believe that data should not be published for web browsing performance using single connection non-threaded page requests. it just isn't accurate at all to reflect common browsing performance. and with connections like telstraclear cable they have a level of burstiness with rate limiting on top. and it's not surprising they come up higher than would in real usage.

i also strongly believe that it's misleading to publish results using graphs that only show the minimum and maximum rather than starting from 0. there isn't that strong a difference - and yet you make it out that there is.

hamish225

1418 posts

Uber Geek

ID Verified

#597434 19-Mar-2012 23:32

Well, for me internal wiring will be eliminated when UFB happens, I'll just connect the modem to my wifi router and phone thing and ill be away laughing.

none of that structured wiring nonsense.

*Insert big spe*dtest result here*

sbiddle

30853 posts

Uber Geek

Retired Mod

Trusted

Biddle Corp

Lifetime subscriber

#597442 20-Mar-2012 06:12

hamish225: Well, for me internal wiring will be eliminated when UFB happens, I'll just connect the modem to my wifi router and phone thing and ill be away laughing.

none of that structured wiring nonsense.

There is no "modem" in the UFB world. You have an ONT that converts fibre to Ethernet, and your touter will connect in to this. Phone services will either connect to the ONT or router depending on how your ISP/RSP wants to provision things.

And good luck thinking WiFi will replace copper, because it doesn't. It is a complimentary technology.