So, as a long time IT worker and good netizen, I decided I would take part in the Commerce Commission's "Measuring New Zealand’s broadband performance" initiative.  The Comcom have partnered with British company, SamKnows, who send you a white box, direct from the UK via Royal Mail.  One side of this box (basically an off-the-shelf consumer-grade router with modified firmware) plugs into your internet router and you plug your Ethernet devices into the other end.

I've had this unit plugged into my home network for about a week.  You can log into a portal that shows your various stats.  So what's the problem?  Well, the problem is that SamKnows' stats just don't align with reality.  On their own FAQ, SamKnows states:

It's really important that the test servers are deployed close to your home (and your Whitebox!) because the further the test server is from you, the higher the latency and the more we have to cross third-party networks, which can distort the results. This is why we operate so many test servers all around the world - locality to you is super important!

Here's my results (via wifi):

• Ping www.geekzone.co.nz - 27 ms
• Ping www.ibm.com - 30 ms
• Ping www.nasa.com - 27 m

The SamKnows portal shows my average latency as 228ms (worst 241 ms, best 225 ms)

The next problem is the reported download speed.  If I plug my PC directly into my router, Speedtest.net shows that I can get in excess of 800 Mbps.  Over many tests, I might get a result of 400 - 500 Mbps, but this is very rare, most results are much faster.  SamKnows is reporting my average download speed as 153 Mbps.  What's worse, is that the box has only found one day where it has decided my traffic is low enough to make an accurate test.

I have a few IoT devices in my home, and it seems the tiny trickle of data that they use to phone home is enough to prevent SamKnows from testing download speed most of the time.  On their site, they state that the test only runs when there is no internet traffic.  Are you kidding?  If you have any smart devices, when is there ever no internet traffic?  They need to set a reasonable threshold, so that if there is only a few Mbps (or even Kbps) traffic, the test will run.

So in summary, it seems obvious to me that:

• The testing servers are a long, long way away, making latency tests useless/pointless.
• Speed tests don't work if you have anything live on your network that talks even a little to the internet
• Speed tests don't reflect national speeds - God only know where the test servers are
• SamKnows portal gives no indication of the location of their test servers

Bill.