Commerce Commission Broadband Report Winter 2020

Forums › New Zealand Broadband › Commerce Commission Broadband Report Winter 2020

1 | 2

Talkiet

4792 posts

Uber Geek

Trusted

#2548431 23-Aug-2020 22:00

ripdog:

Wait what? The only ISP comparison included 3 ISPs and only compared Fibre 100? Why are these reports still so trash? Bring back TrueNet.

The Samknows and Comcom staff involved committed before the start of the program that they would report only when they had sufficient samples of each datapoint to make a reasonably statistically significant analysis.

Truenet had nothing like the rigour around analysis that Samknows have - and wanting them back is the same as wanting bad stats instead of accurate ones.

Cheers- N

Please note all comments are from my own brain and don't necessarily represent the position or opinions of my employer, previous employers, colleagues, friends or pets.

surfisup1000

5288 posts

Uber Geek

#2548442 23-Aug-2020 23:36

Talkiet:

ripdog:

Wait what? The only ISP comparison included 3 ISPs and only compared Fibre 100? Why are these reports still so trash? Bring back TrueNet.

The Samknows and Comcom staff involved committed before the start of the program that they would report only when they had sufficient samples of each datapoint to make a reasonably statistically significant analysis.

Truenet had nothing like the rigour around analysis that Samknows have - and wanting them back is the same as wanting bad stats instead of accurate ones.

Cheers- N

Do you think it is OK that after 2 or 3 years they don't have enough data to report by ISP? Given the amount of taxpayer money they were given.

No stats are not a replacement for bad stats.

Even if samknows suddenly produced an ISP comparison report -- it is going to be based on old data if they are just waiting to reach a cumulative level of data.

Talkiet

4792 posts

Uber Geek

Trusted

#2548444 24-Aug-2020 00:06

surfisup1000: [snip]
No stats are not a replacement for bad stats.

That comment alone is enough to discredit your argument.

N

Please note all comments are from my own brain and don't necessarily represent the position or opinions of my employer, previous employers, colleagues, friends or pets.

bigalow

566 posts

Ultimate Geek

#2548446 24-Aug-2020 00:17

I think New Zealand IPs did very will around the level 4 lock down

bigalow

566 posts

Ultimate Geek

#2548447 24-Aug-2020 00:37

nztim:

I am going to get slammed for this comment (and am prepared for it), but I think fibre max is seriously under priced in New Zealand

for example in the USA GIG is 99USD+TAX which equates to about 150+TAX in NZD, this is a cut throat market we have directly translates to this 10% drop in performance

internet price very by country to country you cant go by 1 country

ripdog

548 posts

Ultimate Geek
Inactive user

#2548455 24-Aug-2020 06:16

Talkiet:

ripdog:

Wait what? The only ISP comparison included 3 ISPs and only compared Fibre 100? Why are these reports still so trash? Bring back TrueNet.

The Samknows and Comcom staff involved committed before the start of the program that they would report only when they had sufficient samples of each datapoint to make a reasonably statistically significant analysis.

Truenet had nothing like the rigour around analysis that Samknows have - and wanting them back is the same as wanting bad stats instead of accurate ones.

Cheers- N

Then this exercise is a spectacular failure and a waste of taxpayer money. None of the stats presented in this report are useful. Comparing technologies is worthless. Averaging latency across the entire industry is worthless. The whole thing is basically "NZ internet is pretty good!"

The fact that they even bothered to include that graph of latency to social media servers is embarassing. What did we learn from that? Snapchat hosts their servers in the US? Wow, what a discovery. Latency to game servers doesn't change between fibre 100 and 1000? Incredible! Who would have guessed? Fibre 100 can stream 4k? Fibre 1000 download rates slightly decrease during peak hours? Incredible research!

The only vaguely interesting graph is the one showing breaking down average download speeds on fibre 1000. Still, without ISP breakdown, it has little value as an actionable data point.

The current situation is untenable. The NZ consumer currently has literally no measured data from which to base their choice of ISP on, and there's no standard for ISPs to reach and compete for. But, Talkiet, don't forget - the consumer never has *no* stats to base their ISP decision on. Lacking any good stats from government, lost and confused souls appear on forums like this one on a weekly basis, looking for advice on choosing an ISP. They choose based on a dozen or fewer anecdotes. There's no worse stats than anecdotes. There's no way this situation is better than what we had with Truenet.

Maybe this situation isn't really SamKnows' fault. Perhaps the ComCom should have required ISPs to pitch white boxes to their customers directly. Whatever the case, this sh!t needs to change.

Hammerer

2476 posts

Uber Geek

Lifetime subscriber

#2548474 24-Aug-2020 07:43

Talkiet:
surfisup1000: [snip]

No stats are not a replacement for bad stats.

That comment alone is enough to discredit your argument.

N

The source of this discussion point was the comment about "bad stats instead of accurate ones". When discussing descriptive statistics we would be better off speaking of "less accurate stats" and "more accurate stats" because the current published states still have some risk, albeit very small, of not describing the actual situation.

It was frustrating that the Truenet stats could fluctuate from report to report but I don't see how it is better to provide no stats. If we can't have some published stats for all ISPs samples then I would like to see some published stats on progress made to reach the minimum sample size for each ISP. That would at least bring to the surface the size of the shortfall and allow us to check progress from report to report.

Equinox IT

Cloud spending continues to surge globally, but most organisations haven’t made the changes necessary to maximise the value and cost-efficiency benefits of their cloud investments. Download the whitepaper From Overspend to Advantage now.

surfisup1000

5288 posts

Uber Geek

#2548487 24-Aug-2020 08:13

Talkiet:
surfisup1000: [snip]

No stats are not a replacement for bad stats.

That comment alone is enough to discredit your argument.

N

OK then, I'll bite. Why do you think it is acceptable to replace truenet with a system that produces no ISP comparison data?

evilengineer

466 posts

Ultimate Geek

#2548492 24-Aug-2020 08:31

I'm sorry, got to say it.

I'll be playing the world's smallest violin for anyone complaining about paying <$100 for a 1 Gig connection and only getting 800 Meg.

First world problem, anyone? 😃

Talkiet

4792 posts

Uber Geek

Trusted

#2548507 24-Aug-2020 09:22

surfisup1000:

[snip]

OK then, I'll bite. Why do you think it is acceptable to replace truenet with a system that produces no ISP comparison data?

In principle, because I believe bad or wrong information portrayed as accurate or authoritative is actively misleading - and is worse than no data.

As a specific, real world example, I offer up the following situation:

A long time ago, in a land not so far away, a BB performance testing agency published a set of test results insisting that a number of RSPs Fibre 100 plans were consistently achieving well over 100Mbps - I think over 200Mbps is some cases. This was clearly wrong - and I knew why, so I challenged the BB testing agency and they doubled down on their claim, even after checking their data.

Of course, the testing was bad and the numbers were wrong. They were using a test file smaller than the burst size used by some bits of the network (deliberate vagueness) which meant that the very small performance tests were completing at a higher rate than 100mbps, but that if the downloads were larger, performance would have been MUCH worse due to the way shapers work.

Only once the error was pointed out to them in a VERY detailed way, and evidence showing their testing was badly misleading, did they change anything - and even then, they didn't change it enough to actually be representative. They also never retracted their misleading results due to a bad test design.

There were also numerous examples where their statistical analyses were terrible - either they analysed things with far too small samples, or they removed data without noting it in the analysis.

So in summary, I think that a test provider that understands networking and stats is better than one that valued splashy statements over accuracy.

Now I do happen to have done postgraduate statistical analysis, and I do happen to have over 15 years in the industry with 10 years deeply involved with BB network design and performance, as well as having been on a number of TCF working parties in related areas. I've been involved with Epitiro, Truenet and Samknows and Samknows is by far and away the best testing provider, largely because Sam is damn smart and as a company they don't make unfounded sensational claims.

Their reports so far have basically said that the Internet is NZ is pretty good - and that's accurate.

When there is a statistically valid comparison (number of volunteers on the right plans, on the right RSPs, in the right locations) they'll do the appropriate reports.

Cheers - N

Please note all comments are from my own brain and don't necessarily represent the position or opinions of my employer, previous employers, colleagues, friends or pets.

OldGeek

896 posts

Ultimate Geek

ID Verified

Lifetime subscriber

#2548515 24-Aug-2020 09:33

I wont repeat the first paragraph here but I have to say that the report lacks the data it sets out to report. If the aim is to provide 'consumers with independent information about internet performance across different providers, plans, and technologies' then it delivers only on the 'plans' (actually 'plan'). The only RSPs reported on are Spark, 2D and Vodafone and then only for Fibre 100 speeds, on page 14.

The rest of the report has no other data relative to 'performance across different providers' so in that respect it fails to deliver on its own stated aim.

Whitebox users can see their own stats via the SamKnows website, and those on a Fibre 100 plan can compare their own stats against the 3 RSPs in the report. I am on a Fibre 100 plan, but being an Orcon (Vocus) user I am excluded from this.

The reality is that no significant effort has been made to get whiteboxes out to the user base. I wonder if, in fact, the 'community' of whitebox users in NZ counts a very high percentage of members being GZers. After all I found out about the whitebox from a GZ forum post. ComCom needs to enlarge this community as a cornerstone requirement to deliver on their aspiration to 'provide consumers with independent information about internet performance across different providers, plans, and technologies' in this report. Don't fix the report to reflect accurate stats on the very small SamKnows whitebox user base in NZ, enlarge the user base to allow accurate stats to be reported as aimed for.

Edit: bad text fixed.

--

OldGeek.

Quic referal code: https://account.quic.nz/refer/581402

surfisup1000

5288 posts

Uber Geek

#2548544 24-Aug-2020 10:22

Talkiet:

In principle, because I believe bad or wrong information portrayed as accurate or authoritative is actively misleading - and is worse than no data.

I agree with that, and haven't said otherwise.

My argument is that it is unacceptable to replace one bad system with another bad system. They've replaced a system that produces poor ISP comparison data with one that produces no ISP comparison data. Consumers are no better off despite millions of dollars of taxpayer money gone to waste.

A primary objective (and most important) of Samknows was to provide 'consumers' with independent information on broadband performance across providers. It is unarguable that samknows failed to meet that objective.

eg, some GZ'ers claim that Spark has an inefficient peering arrangement that results in sub-optimal internet performance in some scenarios.

I don't know if this is true or whether the effect is significant. But with good independent ISP comparison data, it would be evident if that were the case. It means that ISP cost saving measures and subsequent effects on performance are obfuscated from consumers.

Jase2985

13461 posts

Uber Geek

ID Verified

Lifetime subscriber

#2548545 24-Aug-2020 10:22

where are the stats on the number of boxes sent out, and the number that are actively being used? locations etc

how many more do they need to give "accurate stats? there must be data around this somewhere? surely some locations have the correct amount of equipment to give some results?

the comcom/samknows etc should be actively pushing to get more people ad data points because they have had the contract for that long to have produced bugger all, something needs to changes or we are just wasting our money on something we already know.

cbrpilot

955 posts

Ultimate Geek

Trusted

Spark NZ

#2548656 24-Aug-2020 11:52

Jase2985:

where are the stats on the number of boxes sent out, and the number that are actively being used? locations etc

how many more do they need to give "accurate stats? there must be data around this somewhere? surely some locations have the correct amount of equipment to give some results?

the comcom/samknows etc should be actively pushing to get more people ad data points because they have had the contract for that long to have produced bugger all, something needs to changes or we are just wasting our money on something we already know.

Suggest you read page 4 of the report. That is where they need more volunteers.

No idea what the comcom are doing to encourage more people to sign up as trialists.

To be fair (esp for ADSL and VDSL) they are going to be a challenge to maintain as more and more people move to UFB they will have to keep finding new ADSL and VDSL participants.

My views are my own, and may not necessarily represent those of my employer.

1 | 2