Geekzone: technology news, blogs, forums
Guest
Welcome Guest.
You haven't logged in yet. If you don't have an account you can register now.


quic

236 posts

Master Geek
+1 received by user: 754

Trusted
Quic Broadband
Lifetime subscriber

#311025 7-Dec-2023 18:49
Send private message

Hi all,

 

 

 

I just wanted to put a quick message out to you to give my heartfelt apologies for the disruption our network issues have caused you today. I’m extremely aware of the pain felt “on the ground” by you, and the grief caused to you.

 

I also want you to know I have been hearing the feedback that’s come through today and this is being taken on board.

 

 

 

Today has been an absolute nightmare, and while we’ve been meticulously planning the migration to our new network for around 6 months now, working through every detail, ultimately today has shown this has failed.

 

(this has been part of the reason why we have been quiet here - not because we've hopped on a ship to Mars, we have seriously been all heads down to get this new network up and running and move all of you across, leaving any packet loss behind)

 

 

 

Why are we doing this? Our network has been operating based on the initial build around 8 years ago, and has stood the test of time, however with the growth of our business and network, it’s starting to show it’s age as you all are very aware. With evolutions in networking technologies, we need to invest in a new network to fuel the next stage of our growth and provide you with better services, more reliability and more resiliency.

 

 

 

The team are absolutely exhausted, gutted, and believe me – just as frustrated as you are, if not more, to have been forced to roll back to our existing network today.

 

 

 

Today has been full-on firefighting and working to restore all services, so I don’t have many answers to share to the questions you all have at the moment, over the coming days and weeks before Christmas, we will be fully completing an in-depth analysis to determine exactly what went so wrong today, including engaging third party consultants, our networking vendor, debriefs, and ultimately reviewing our plans for our new network to ensure we’re both building it in the right way, and our migration strategy is solid.

 

 

 

As a result of this, an incident report will become available once this exercise has been completed. Please let me know if you wish to receive a copy by sending a DM here with your email address (matching that on your account) and I’ll ensure this is done.

 

 

 

For now, all I can give you is an honest apology until I have more to share once we’re out of “recovery mode”, and a promise that I will be personally ensuring all of the above actions to ensure that we get the answers we need to ensure that our network can continue to be at the forefront to fully support your telecommunications requirements.

 

 

 

Many Thanks for your ongoing support,

 

Shaun

 

 





Quic Broadband
quic.nz - The telco who puts you in charge!

 

 

 

 


Create new topic
Linux
12173 posts

Uber Geek
+1 received by user: 8469

Trusted
Lifetime subscriber

  #3169413 7-Dec-2023 18:56
Send private message

@quic straight up and honest no one could ask for more - I take my hat off to you and the team and I am even not a customer (yet)




michaelmurfy
meow
13579 posts

Uber Geek
+1 received by user: 10910

Moderator
ID Verified
Trusted
Lifetime subscriber

  #3169424 7-Dec-2023 19:46
Send private message

Thanks so much Shaun.

 

I’ve been there and done that with firefighting changes and totally understand how it is. I’ve got a backup for work and so just continued working today trying to not bother you. Seriously, hats off here with your straight up honesty. With any other ISP I would have left them in a heartbeat but it’s these updates and the human touch to things that actually caused me to fully move across - problems and all.

 

I have complete faith you’ll get this sorted. I know the frustration, many of us have been there and done that through the years. I’ve been involved and even once caused production outages in the past that have been in the media and have been through production incidents that have caused me to work until the corners of my monitors started dancing. I feel everyone needs to work in this space to gain an understanding on just how difficult some production changes can be.

 

As I’ve said many times today thankyou for your attempt and also thankyou for having the courage to both try and roll forward and make the difficult call to roll back. Just have a breather, feel free to reach out if there is anything I can do to assist and I look forward to round 3 😊





Michael Murphy | https://murfy.nz
Referral Links: Quic Broadband (use R122101E7CV7Q for free setup)

Are you happy with what you get from Geekzone? Please consider supporting us by subscribing.
Opinions are my own and not the views of my employer.


aj6828
177 posts

Master Geek
+1 received by user: 47

ID Verified
Trusted

  #3169431 7-Dec-2023 20:43
Send private message

thanks for being transparent this things happen hopefully everything will be sorted soon :) 





Exclusive for Geekzone Members!
Dynamic IP & Bring Your Own Device Hyperfibre & Fibre plans & NO Contracts with Hyperline.co.nz powered by ASN 9790 Network

 

Chrous areas only HYPERLINE.co.nz  




Johnk
837 posts

Ultimate Geek
+1 received by user: 192

ID Verified
Trusted
Lifetime subscriber

  #3169434 7-Dec-2023 21:05
Send private message

And again, posts like this are why I still give you my money each month instead of moaning.

Keep up the great mahi you and and the team are doing.

Thanks

Lias
5655 posts

Uber Geek
+1 received by user: 3978

ID Verified
Trusted
Lifetime subscriber

  #3169435 7-Dec-2023 21:21
Send private message

Appreciate the transparency as always.

 

Can't lie, this morning was frustrating as heck, but having been one of the engineers under the pump as your meticulously planned change goes sideways before you have my sympathy.

 

 





I'm a geek, a gamer, a dad, a Quic user, and an IT Professional. I have a full rack home lab, size 15 feet, an epic beard and Asperger's. I'm a bit of a Cypherpunk, who believes information wants to be free and the Net interprets censorship as damage and routes around it. If you use my Quic signup you can also use the code R570394EKGIZ8 for free setup. Opinions are my own and not the views of my employer.


Chippo
129 posts

Master Geek
+1 received by user: 52

Trusted

  #3169438 7-Dec-2023 21:56
Send private message

Thanks Shaun and @quic team,

 

For those of us that're still offline - with active PPPoE but no routing, even after multiple restarts - what's the best contact method to point you to us?

 

 

 

 





I work for a global Data Protection Software company - But my opinions are my own.


 
 
 

Stream your favourite shows now on Apple TV (affiliate link).
cjkbarnett
37 posts

Geek
+1 received by user: 18


#3169440 7-Dec-2023 22:17
Send private message

Got the odd Hetrix Tools "My Server is down" email here and there today, but came back up pretty quick.

Unfortunately, internet died for good tonight at 8.48pm, just as a i started war thunder :(

Done the normal check Dns, give it all a good restart, but no luck. Nevertheless, plug phone in and start tethering, and away we go.

Appreciate the status updates and transparency, and I get that these things happen. Best of luck to team Quic!

michaelmurfy
meow
13579 posts

Uber Geek
+1 received by user: 10910

Moderator
ID Verified
Trusted
Lifetime subscriber

  #3169521 8-Dec-2023 08:50
Send private message

@Chippo Log a fault in the Quic portal with that in the notes and they'll take care of it.





Michael Murphy | https://murfy.nz
Referral Links: Quic Broadband (use R122101E7CV7Q for free setup)

Are you happy with what you get from Geekzone? Please consider supporting us by subscribing.
Opinions are my own and not the views of my employer.


xpd

xpd
Geek of Coastguard
14115 posts

Uber Geek
+1 received by user: 4574

Retired Mod
ID Verified
Trusted
Lifetime subscriber

  #3169650 8-Dec-2023 10:47
Send private message

Thanks for the update, apart from upsetting the wife's workflow yesterday (hey, it was either going to be me unplugging something or you :D ),  I'm happy with the service and open door policy you have with info etc :)

 

I'm staying put for now :)

 

 





XPD / Gavin

 

LinkTree

 

 

 


hsvhel
1273 posts

Uber Geek
+1 received by user: 596

ID Verified

  #3169717 8-Dec-2023 12:25
Send private message

Whilst it was an inconvenience, some of the other thread comments could well do with supporting the smaller ISP instead of the flailing hands that were going on.

 

We all signed up knowing this sort of thing would happen with expansion.

 

On a positive note, with an enforced technology break I got to throw a leg over.....Happy Friday lol

 

Bright side, there is always one if you look for it

 

Chur Quic, you'll get it sorted

 

 





Referral Link Quic

 

Free Setup use R502152EQH6OK on check out

 

 


gehenna
8667 posts

Uber Geek
+1 received by user: 3883

Moderator
Trusted
Lifetime subscriber

  #3169718 8-Dec-2023 12:25
Send private message

A lot of providers in NZ could learn from this approach. 


 
 
 
 

Shop now for Dell laptops and other devices (affiliate link).
3l3m3nt
120 posts

Master Geek
+1 received by user: 152

ID Verified
Trusted
Lifetime subscriber

  #3169827 8-Dec-2023 16:50
Send private message

I know I'm just echoing the sentiments shared by others here, but as much as it probably hurts to be defeated a couple of times, in my experience it makes the victory much more sweeter!

 

Best of luck for the next attempt. I'll still be here :)





Dunedin, NZ
Quic Broadband  | Rocket 1G/1G Hyperfibre (Yes, you read that right!)
Dunedin Live Webcam (4K) | Quic Smokepings

 

Referral Links:
Quic (use R282731EPGJMG on checkout for free setup, and to help me pay for my fast internet addiction)
Contact Energy (use FRTDD2R for $100 credit)


mentalinc
3384 posts

Uber Geek
+1 received by user: 1023

Trusted

  #3173215 18-Dec-2023 20:00
Send private message

Hope the PIR went well and there was some nice clarity provided from the vendors involved.

 

Do we have an ETA for when the change over will be attempted next (I assume it's a Jan thing now with the change freezes often in place mid last week)? or is it still a case of working through the issues identified to find resolutions? 





CPU: AMD 5900x | RAM: GSKILL Trident Z Neo RGB F4-3600C16D-32GTZNC-32-GB | MB:  Asus X570-E | GFX: EVGA FTW3 Ultra RTX 3080Ti| Monitor: LG 27GL850-B 2560x1440

 

Quic: https://account.quic.nz/refer/473833 R473833EQKIBX 


Create new topic








Geekzone Live »

Try automatic live updates from Geekzone directly in your browser, without refreshing the page, with Geekzone Live now.



Are you subscribed to our RSS feed? You can download the latest headlines and summaries from our stories directly to your computer or smartphone by using a feed reader.