Hi all, 👋

I'd like to make a public post in this strongly valued community around the issues our customers may have experienced with Quic lately, what's been happening, and what's being done about it.

We hugely value everyone in our Geekzone community - you are the users who report valid issues to us, provide us with real-world insights, grant us access to data to help improve the network, and are always first to pop hands up to help us in any testing required.

Being open, honest and transparent (within commercial realities), we believe, is key to our success in this community - and is why I'm posting this publicly, to share these insights to a very tech-savvy audience who we know values open communication.

In short, this post very much has the feel of us putting our heart on our sleeve.

To start with, thanks in large to this community, Quic has grown at a rapid pace. While I can't share specific figures, we're talking around a 20-25% month-on-month growth, and substantial growth in network traffic, an example being in the screenshot below looking at a single link within our network over 3 months - which helps give an idea of the growth we're seeing.

This doesn't mean our network is congested - none of our links are - however, keeping this pace requires changes and upgrades, which we're working on currently.

While Quic operates under the Vetta network, we do operate as our own entity, and operate our own infrastructure which is dedicated to Quic, and thus is a subject of its own challenges and keeping up with Quic's network growth.

This has been a conscious decision, as while we could have entered the market strictly reselling another providers services, we didn't want to go down this approach and wanted to stand on our own two feet - which we'll continue to do more and more as time goes on.

It's obvious and goes without saying that we've had some growing pains, including everything from software issues, hardware issues, and issues that come along as we start seriously scaling (e.g. the network issue from this morning being in part to due an SQL database locking).

While we have been working on upgrades in the Quic network, unfortunately with networks of any decent size, especially ISP networks expected to work 24x7, upgrades to take time to plan and implement, while trying to reduce impact to the production network.

With this in mind, I want to give a small insight and share what we've been doing, and what's happening going forward in the Quic network. These are in no particular order of priority.

 

1. Network Operations team - We have recruited additional (well experienced) staff into our network operations team to further advance the Quic network and increase the pace that we can complete upgrades and strengthen our network.
2. System upgrades & IPoE - Our development team has been working over the last 3 months, and have almost completed a full rewrite of our authentication and accounting systems, which will support both PPPoE & IPoE/DHCP dual-stack, using a significantly higher performance database engine behind the scenes. This new rewrite will also provide you with even more insights on your connection than what is provided now.
3. Network upgrades - We are hard at work commissioning new BNG's and network devices alongside our system upgrades from point 2, which will not only support the changes to our network required to bring in support of IPoE/DHCP, but are also higher performance devices with higher capacity to cater for future growth.
4. Additional LFC links - We are also starting the process of additional handover links with our LFC partners (e.g. Chorus). While we don't strictly require these just yet, we're wanting to get ahead of the curve.
5. IGP & core router changes - To rectify the growth issues we have experienced in relation to resilience & incident recovery time, and packet loss, plans are in place to perform changes to both our IGP routing within our network, and changes to vendor software on the routers and network elements we operate, which are better suited to the network we now run.
6. Automation improvements - We're also working on additional automation improvements in our stack. The end goal is customers will be able to completely self-service provision connectivity to ONTs around the country.
7. Now that we believe we have targeted the tech-savvy audience correctly, we are looking at ways that we can better offer support to our customers. While we are a self-service ISP, we recognise there are times that you need to communicate with us, from a provisioning issue, to a second pair of eyes. We’re testing a variety of approaches to this (some of which you may have seen), but we need to be careful to get the balance and access right between valid technical issues to a tech-savvy individual, and providing full support – which we’re not set up for.

In the coming weeks and months, we will be putting calls out to beta testers to assist us in real-world testing of some of the above points, and you'll also be seeing some maintenance on our status page to implement the above (if you're not subscribed, I encourage you to look at doing so - https://status.quic.nz)

Our three largest priorities are points 2, 3 & 5 above. These are the changes you should expect to see first. Without committing to timelines, as an example, we're expecting to be able to start testing our new IPoE support in the next couple of weeks - watch this forum!

In short, we've started small, and are now rapidly growing - and heavily investing into our network now that we have reached both scale and commercial viability.

Quic has been built from the ground up to be a new breed of self-service ISP, offering unmatched insights into your connection and ONT details, and the power to self-troubleshoot. However, we are still a small ISP and have been a victim of some "chicken and egg" scenarios of what we can complete, and when we can complete it, as we roll through the growth of our business at this startup phase.

For those customers who we've caused interruption to, on behalf of the team, we are genuinely sorry - this is of course not the service we want to provide, and rest assured we're not ignoring or taking lightly any impacts which have been caused. The commitment I can give you is that we are working hard to increase the resilience, fault tolerance, and performance of our network.

For those customers who are eager to help us in any way possible, please keep an eye out here in this forum for a call for beta testers - this isn't far away, and I can't wait to share our improvements with you.

As a thank you to our customers for shouldering some pain of issues experienced, I'm pleased to announce that we are postponing price increases of our plans for existing customers for one month - moving to 1st November. This means we will shoulder the additional cost of increased plan prices from the LFCs for an extra month, as a small gesture to say thanks for being along this massive wave with us.

I'm looking forward to sharing more of our upgrades with this community as they come to fruition, and continue to improve on our goal to be the go-to self-service ISP for the tech community.

🧡