Samknows is coming... and this is what it looks like.

I attended the two hour presentation at the Commerce Commissions on Samknows today in Wellington. It was a brilliant presentation from Sam and Roxanne from Samknows and Simon from the Comcom.

I've asked to get a copy of the slide-deck but here are the highlights from the notes I took from the presentation.

Comcom are looking for around 3000 signups across the major RSPs in NZ and that is also looking to include Wireless Broadband offerings from Skinny, Spark and Vondafone (I think it was these 3 and 2D wasn't included as they don't offer WBB). Looking to have probes across all the major access types (ADSL,VDSL, UFB100/20, UFB GB & Vodafone HFC).

I did raise the situation of BUBA customers and the poor Conklin folks as they are a very vocal lot and Chorus from my understanding aren't planning to deal with them as all the low hanging fruit has already been hit it will be interesting to see what the Comcom and Samknows do about finding them. Or if they do find them how the ADSL service should be differentiated between BUBA aka ADSL1 vs EUBA aka ADSL2. I plan to have further discussions with the Comcom about that. If Samknows have access to all the Chorus copper address data it's easy to find out who's BUBA and who isn't.

I'm a bit unsure of how the Wireless Broadband probe testing is going to work out as the Samknows probes could transfer a whole lot of data on a 80Mbit or higher Mobile Data connection over a 10 second test. In my view unless some deal is struck with the Mobile telcos with the data being zero rated I am not sure how much data it will be consuming, back of the envelope it could be blowing a fair bit of the 120/240GB data plans.

Also I didn't raise questions (but probably should have) around WISPs around the country. Then again those numbers would be relatively small.

I made the recommendation to say a condition of signing up to a probe on a Fixed Broadband probe the panelists really should be on unlimited as all the ISPs offer unlimited.

The switch (not router) plugs into your home broadband and isn't doing NAT but monitoring traffic flows from the client side to the "wan" port to make sure that no significant usage was occurring.

• To me this is major since there will be no double NAT between wired connections and the LAN port on the router.
• The probe also include a wireless "monitor" that doesn't run as an AP but as a wireless client. It does clever detection do find the internet connections wireless network by taking the default ports MAC address and then comparing it to the AP's BSSID/MAC and doing a +/- 2-3 numbers to "guess" which is the SSID of the router and it also scans both 2.4G and 5G radios. I was fairly impressed with that as a way to find the wireless network as I was asking questions around how they would determine which wireless network was in use in densely populated areas.
• It also monitors the WAN port and if other MACs are detected from ARPs on the network other than the default route then it knows the user has other devices plugged into the LAN switch not behind the probe so notifies the customer they should have all their devices connected behind the probe.
• They were clear they don't do any passive monitoring of client traffic, only use the client side to decide if they are going to run a test or not based on lack of any throughput of data.

They run true speed tests in the probe itself rather than downloading a fixed file size which is an 8 threaded test that has a ramp up of 2-3 seconds and runs a 10 second saturation test. For anyone who has done this sort of testing will know this is VASTLY superior to TrueNet's testing of a fixed size and reason in itself IMHO that the Comcom made the right decision to go with Samknows rather than over Truenet.

They do continual jitter and latency tests of around 2000 packets per hour to all their probes across the world. So while it doesn't indicate speed it does indicate when there are latency spikes due to network events.

The test a myriad of different services including streaming services. For video streams they actually download the streams and as long as the stream stays ahead of the time. First retrieve a popular video in NZ. Then connect to the local CDN and start streaming the video itself. And check the video metadata for the time offset at 10 seconds or so arrived before 10 seconds have elapsed then they know that there wasn't buffering of the video. Also was suitably impressed with that.

They do tests hourly or on whatever schedule they want. For wireless tests they check the air is clear, if it's not wait another 30 seconds and keep on waiting for around 5 mins before giving up for that hour.

Sam said typically they loose 10-15% of tests per hour due to the internet being active at the time. Which considering peak hours I thought was surprisingly low.

There wasn't any plans to include the smaller ISPs. The view was if the ISPs wanted to give Samknows money for the supply of the probes then they could be included in the stats, but wasn't something that the Comcom planned to fund.

The probes testing scheme is to talk back to national testing probes that would be connected to peering neutral IXs. So Sam expected to have dedicated nodes in Auckland and Wellington IX. They did have options for probes within the ISPs outside NZ but said there was only plans to have neutral probes on the IXs. So this would mean Spark and Vodafone would have to peer. Which I found rather interesting.

There was discussion about if the Comcom will supply the data to everyone and in what form. Samknows view was "it's the Comcom's data and they can do with it what they want" and the Comcom view was "we are looking into all options" so they were fairly open to the ideas of Open Government APIs or whatever works.

Numerous references to the Ofcom (UK) and FCC (US) regulators and how everything differs per geographical region. As Sam said, they have been doing this for 10 years so have a lot of experience in this space. I found it interesting that the FCC release all the source data every month as a 5GB data download, lets see if the Comcom follow suit or what happens in that space. 

I asked a few questions around on-boarding of customers especially in the ADSL space where I asked if they planned to use Chorus data to see if people were getting a terrible connect rate (due to wiring issues etc and should get a Master Filter aka Splitter) and if as part of the onboarding they made efforts to get the best speed possible for the DSL customers. Not something they were specifically planning to do only unless someone was a real outlier. Something I plan to take up further with both the Comcom and Samknows as IMHO it should be done as a matter of course.

End customers will get a customer dashboard, and the Comcom will have a single all of country view which looks pretty even though they only had screenshots and didn't demo it.

Bit of talk about figuring out what sort of traffic was being generated. But the general concensus was that Samknows, The Comcom and the RSPs were going to be fairly open and honest about where the probes are as in reality there isn't much an ISP can do to "game" the system short of grooming customers off handovers if they know some handovers are congested and they don't want to have bad stats at peak time. The interesting thing was there was talk about having a second set of probes talking to different end points as a "shadow" system to make sure the ISPs weren't gaming it. To me this was also a major departure from how Truenet operated as they were actively hostile to letting my ex-employer know who the probes were and if they were having specific issues.

Over all it was a very informative and excellent talk. It was clear that Sam knew his stuff.

Roll on Samknows !! http://www.comcom.govt.nz/broadbandvolunteer