25/9 Quic down

Forums › Vetta and Quic › 25/9 Quic down

1 | 2 | 3

meow
13580 posts

Uber Geek
+1 received by user: 10912

Moderator

ID Verified

Trusted

Lifetime subscriber

#3131771 25-Sep-2023 21:16

Linux:

SomeoneSomewhere:

Yup, back up. Surprised it didn't end up on the status page.

Would you rather they fix the issue fast or worry about a status page update?

It is also South Canterbury Anniversary Day today so they're on holiday. Likely a single on-call person who had to login and resolve.

Remember the sticky too: https://www.geekzone.co.nz/forums.asp?forumid=194&topicid=307025 - Quic have had some teething trouble but are actively working through these.

Michael Murphy | https://murfy.nz
Referral Links: Quic Broadband (use R122101E7CV7Q for free setup)

Are you happy with what you get from Geekzone? Please consider supporting us by subscribing.
Opinions are my own and not the views of my employer.

nic.wise

333 posts

Ultimate Geek
+1 received by user: 36

Trusted

#3131776 25-Sep-2023 21:28

No issue here and mine is normally quick to complain if it’s down for even a few seconds.

Nic Wise - fastchicken.co.nz

kennedybaird

63 posts

Master Geek
+1 received by user: 50

ID Verified

#3131778 25-Sep-2023 21:51

Linux:
SomeoneSomewhere:

Yup, back up. Surprised it didn't end up on the status page.

Would you rather they fix the issue fast or worry about a status page update?

For me, status page update. Always.

Updating a status page is a known quantity of time (or certainly should be). You don't always know how long a problem might take to fix.

They also serve as a historical method of transparency as well

Lias

5655 posts

Uber Geek
+1 received by user: 3978

ID Verified

Trusted

Lifetime subscriber

#3131887 25-Sep-2023 23:13

Didn't notice cos I was busy playing Starfield, but yep smokeping says it was down for me too.

I'm a geek, a gamer, a dad, a Quic user, and an IT Professional. I have a full rack home lab, size 15 feet, an epic beard and Asperger's. I'm a bit of a Cypherpunk, who believes information wants to be free and the Net interprets censorship as damage and routes around it. If you use my Quic signup you can also use the code R570394EKGIZ8 for free setup. Opinions are my own and not the views of my employer.

allan

2073 posts

Uber Geek
+1 received by user: 899

ID Verified

Lifetime subscriber

#3131928 26-Sep-2023 08:07

Linux:
SomeoneSomewhere:

Yup, back up. Surprised it didn't end up on the status page.

Would you rather they fix the issue fast or worry about a status page update?

Having spent a good chunk of my life working in IT Service Delivery, status page update every time - saves a zillion calls to the help desk (assuming you have one)

ibewaffle

29 posts

Geek

#3131935 26-Sep-2023 08:37

My connection never came back up :( Have tried powering off everything, waiting a few minutes then powering back on.

GPON is on and LAN1 is blinking but nothing. Tried connecting directly with my Mac and creating a PPPoE service but still nothing there. Lodged a fault with Quic as it only happened with the outage last night so assume it must be something funky with my connection.

Mighty Ape

Shop now at Mighty Ape (affiliate link).

hsvhel

1273 posts

Uber Geek
+1 received by user: 596

ID Verified

#3131963 26-Sep-2023 10:43

The timing of this one was gold.

Mrs was telling the kids to turn off and get ready for bed, or else the internet will stop

Boom.....internet stopped (breifly).

Some hand of god action!!

Referral Link Quic

Free Setup use R502152EQH6OK on check out

xor

75 posts

Master Geek
+1 received by user: 12

#3135692 28-Sep-2023 00:58

Linux:
SomeoneSomewhere:

Yup, back up. Surprised it didn't end up on the status page.

Would you rather they fix the issue fast or worry about a status page update?

So they only have a single person working in the entire company? Or does a network-wide outage not justify managerial involvement?

hsvhel

1273 posts

Uber Geek
+1 received by user: 596

ID Verified

#3135711 28-Sep-2023 07:59

@xor
Just a single person, but they are part time with split personalities so you get about 4
In total

Referral Link Quic

Free Setup use R502152EQH6OK on check out

nic.wise

333 posts

Ultimate Geek
+1 received by user: 36

Trusted

#3135713 28-Sep-2023 08:11

hsvhel: @xor
Just a single person, but they are part time with split personalities so you get about 4
In total

Not exactly uncommon on a long weekend tho. One person on, they escalate if needed.

Did exactly that with a multi-billion dollar payments platform… (it was 2 people cos we ALWAYS had 2 on call, but that was just our model). Could escalate to anyone in the company if we thought we needed to.

Nic Wise - fastchicken.co.nz

Linux

12182 posts

Uber Geek
+1 received by user: 8476

Trusted

Lifetime subscriber

#3135981 28-Sep-2023 15:30

@xor Do you know how many people work for company?? I rather they fix the root of the issue ASAP than update the website before looking into the issue

Engineers that know how to fix major outages are not going to go oh the website is not updated we better not do anything before hand!

I worked in Telco land for 17+ years and updating the website when crap hit the fan was the last thing on our mind!

Geekzone support

Support Geekzone with one-off or recurring donations Donate via PressPatron.

michaelmurfy

meow
13580 posts

Uber Geek
+1 received by user: 10912

Moderator

ID Verified

Trusted

Lifetime subscriber

#3135990 28-Sep-2023 15:44

@Linux It is a small company but they have really talented engineers. A few of us here are involved in testing and are in close communication with people there. Not going to give indication of numbers.

But I agree. In the event of an outage the first point of call is diagnosing and fixing. The incident manager is the one who normally would have the pleasure of providing updates to stakeholders, that is never the engineers responsibility. In the event of an on-call situation where you'll likely have one engineer (and a secondary, if required) communication falls on best effort with all effort going to fixing the issue instead. I've done many years in incident management and have been on the end of fixing major incidents for the biggest bank in NZ and I must say if internet banking is down and you've got a million customers all screaming at everyone to get it fixed my priority is to get this fixed and just provide updates when I have them but normally in the first half hour of a major incident communication is rather limited and engineers are given space.

This wasn't a full outage, it was partial and it didn't last long. My own connection didn't go down but a friends did plus also I am not kidding when I state it was actually a public holiday for their team during this outage so on-call would have been the one involved.

SomeoneSomewhere

1882 posts

Uber Geek
+1 received by user: 1086

Lifetime subscriber

#3136006 28-Sep-2023 16:30

True - but also, a "we are aware that we are experiencing issues and a looking into it" posted to the status page is something that should not be onerous. It's not a ten minute phone call that you're trying to get off; it's not writing/answering emails; it should be half a dozen clicks to post a pre-written announcement. That's what the status page is for.

hsvhel

1273 posts

Uber Geek
+1 received by user: 596

ID Verified

#3136007 28-Sep-2023 16:32

Ahhh the service industry, just can win with all the crowd......

Referral Link Quic

Free Setup use R502152EQH6OK on check out

RunningMan

9186 posts

Uber Geek
+1 received by user: 4840

#3136051 28-Sep-2023 17:44

michaelmurfy:[snip] In the event of an outage the first point of call is diagnosing and fixing. The incident manager is the one who normally would have the pleasure of providing updates to stakeholders, that is never the engineers responsibility.

That's an engineer's perspective of the problem solving process, however any business should be looking at issues from the customer's perspective too; after all it's them keeping that company in business. Not saying the engineer's view is wrong, but keeping paying customers informed is a key part of any business, and in these times of quick and easy to diseminate (social) media it should be set up to be a simple task for somebody to do.

In the case of an ISP, then as soon as an issue is recognised that is likely having significant impact on multiple users then a status update via the perferred method (status page, social media etc.) should be issued. The company can have processes which work for them so whether it's an engineer escalates to someone, or has authority to hit go on a preformatted generic message themselves. The guidlines for what is significant and what impact it is having can be drawn up in advance, but generically it would be reasonable for total loss of service or loss of a key part of the service (e.g. DNS) for a suburb or bigger area to reach this threshold.

It doesn't need much or any detail at all - as simple as something's gone wrong, we know about and are looking at it would work. This may take a minute of somebody's time, but may save hours of time for 100s of customers, as well as just demonstrating a customer focus (which customers generally care about quite a bit, especially when there's an issue).

@Quic's recent post about broader issues and their resolution really hits the mark with how good customer communication can work - it's helpful to those troubleshooting what could be local network issues as well as really showing a customer mindset, and really refreshing to see. Presumably they have some sort of internal company portal, and particularly with all the automation side of things they are doing, it should be reasonably easy to have a button that sticks a time stamped post on the status page that something's wrong and we'll look at it. An on call engineer on a public holiday ahould have available to them a simple process to activate a basic status update (either personally or escalation) that they can do without significantly removing them from the primary task of fixing the fault.

1 | 2 | 3