Micro outages last night ?

Forums › Vetta and Quic › Micro outages last night ?

1 | 2 | 3 | 4 | 5 | 6 | 7

5655 posts

Uber Geek
+1 received by user: 3978

ID Verified

Trusted

Lifetime subscriber

#3169305 7-Dec-2023 13:06

Sinuation:

Back up now too. Dang another rollback. I really hope they do this at a Friday next time.

Hell no.. I do not want to spend a Saturday with grumpy kids with no youtube

I'm a geek, a gamer, a dad, a Quic user, and an IT Professional. I have a full rack home lab, size 15 feet, an epic beard and Asperger's. I'm a bit of a Cypherpunk, who believes information wants to be free and the Net interprets censorship as damage and routes around it. If you use my Quic signup you can also use the code R570394EKGIZ8 for free setup. Opinions are my own and not the views of my employer.

prat33k

182 posts

Master Geek
+1 received by user: 31

#3169306 7-Dec-2023 13:12

Interesting routing change with rollback though..

Now to reach an IP within AKL (behind compass network) - it is routing me via Sydney now instead of local peering.

I will have the secondary connection live for now until they complete the migration. Hopefully, they attempt before the end of the month as well :)

t0ny

414 posts

Ultimate Geek
+1 received by user: 84

Lifetime subscriber

#3169308 7-Dec-2023 13:20

Fixed for me in Auckland however, has to be said that Quic really needs to step up here if they want to build up on customer loyalty. A lot of us work from home or have people working from home hence a reliable connection is extremly important. Will be looking forward to the RCA when it comes out.

Sinuation

151 posts

Master Geek
+1 received by user: 73

ID Verified

#3169312 7-Dec-2023 13:41

t0ny:

Fixed for me in Auckland however, has to be said that Quic really needs to step up here if they want to build up on customer loyalty. A lot of us work from home or have people working from home hence a reliable connection is extremly important. Will be looking forward to the RCA when it comes out.

Have to agree here. It rolling into early business hours is too much. And all that grief only for it to result in a rollback hurts.

AgentK

34 posts

Geek
+1 received by user: 11

#3169313 7-Dec-2023 13:41

"Fixed" in a sense that there is a general connection, sure.

The latency and the routing is still bad, much higher than what it used to be before the migration started.

Imho, they should have made the decision to rollback their migration at 6:30am or 7am and not continue for as long as they did and the overall process and communication has been a shit show - small team or not.

Will be interesting to see what their post-mortem is gonna look like and if they actually make it public. But this incident and its handling have made it very, very obvious that there's a lack of maturity in their organisation and how they deal with customers.

I have very little trust that they'll be able to pull this off when they try for the 3rd time - might just bite the bullet and move on asap and maybe have another look at them in 6-12 months.

michaelmurfy

meow
13581 posts

Uber Geek
+1 received by user: 10914

Moderator

ID Verified

Trusted

Lifetime subscriber

#3169314 7-Dec-2023 13:42

Sinuation:

Back up now too. Dang another rollback. I really hope they do this at a Friday next time.

Friday is worse IMHO. It didn't worry me too much as I did have a few backups (another UFB for now but also 5G).

I know everyone is frustrated but I've come from the networking / site reliability engineering space and have first hand experience with these changes. I went almost for a week dealing with a large scale DDOS and standing up mitigation for ANZ a few years back https://www.nzherald.co.nz/business/cyber-attack-anz-online-access-down-for-third-day/QI2VYU3RNN6JPR2TACF57DSYUI/ and that really took its toll and actually took me at-least another week to recover - during that time I had customers, staff and all hounding me, sending me abuse and horrible messages and all while I was stuck in meetings for often 12 hours at a time. We got there, and I am very proud of what I helped implement (which is still used today) to protect ANZ from future outages like this but it is those messages that are not at all helpful.

It is OK to be frustrated, you're also not contracted to stay with Quic but seriously I have full respect for the team there attempting a roll forward even if they had to stop and roll back all their changes. That to me shows excellent change management processes that they can roll back in a matter of minutes and just shows how agile they are.

No doubt the next round they'll have more of an understanding of the potential problems - they do have dev and beta (a few of us) to play around with until then but from what I saw today for the small times things did work as expected it was pretty awesome seeing the substantial drop in latency to some destinations and zero packet loss.

You can never understand how complex it is to replace an entire core network until you've experienced it. You all are experiencing it all and also experiencing their frustration too but be kind and #hugops. I'm looking forward to the new network being fully implemented.

Michael Murphy | https://murfy.nz
Referral Links: Quic Broadband (use R122101E7CV7Q for free setup)

Are you happy with what you get from Geekzone? Please consider supporting us by subscribing.
Opinions are my own and not the views of my employer.

Dyson

Shop now for Dyson appliances (affiliate link).

kennedybaird

63 posts

Master Geek
+1 received by user: 50

ID Verified

#3169331 7-Dec-2023 14:01

Yeah, totally understand.

For me it's just a challenge as I'm syncing with US, so most of my meetings are early morning.

And today, finally getting into the swing of work and it's randomly dropped out again.

I do feel like it's a little unfair that we're paying full price for such an intermittent service. It's also well and good to say "just change to another provider" - but that ends up being a whole other set of brain cycles

sud0

283 posts

Ultimate Geek
+1 received by user: 117

ID Verified

Lifetime subscriber

#3169332 7-Dec-2023 14:03

michaelmurfy:

I went almost for a week dealing with a large scale DDOS and standing up mitigation for ANZ a few years back https://www.nzherald.co.nz/business/cyber-attack-anz-online-access-down-for-third-day/QI2VYU3RNN6JPR2TACF57DSYUI/

I think that's the same event that took away some of the biggest customers from RedShield (i was at RS back then). It was quite a mission.

Lucas

lpossamai.me

kennedybaird

63 posts

Master Geek
+1 received by user: 50

ID Verified

#3169333 7-Dec-2023 14:03

That being said, absolutely can't imagine how much of a bad day the team are having - up all night, it takes longer than expected, probably just taking a break then need to come back.

Been there before with deployments and it is NOT fun discovering an issue when you think you're finished

Sinuation

151 posts

Master Geek
+1 received by user: 73

ID Verified

#3169334 7-Dec-2023 14:12

I agree it must be more than a bad day at the quic team. Michael's insight is always great.

Have to also agree with Kennedy regarding the paying the full price for intermittent connections, especially during work days. And yeah changing providers on the drop of the hat is a bit of a pain, especially doing it during working hours.

Still, I have faith the Quic team will get this sorted. Even if it's a bit of a pain. Though I would like some sort of communication from them, if that's not too much.

naggyman

697 posts

Ultimate Geek
+1 received by user: 86

#3169335 7-Dec-2023 14:15

Seem to be completely down now, other than services being served from my local Cloudflare POP.

Morgan French-Stagg

morgan.french.net.nz

Lego

Shop now for Lego sets and other gifts (affiliate link).

t0ny

414 posts

Ultimate Geek
+1 received by user: 84

Lifetime subscriber

#3169336 7-Dec-2023 14:33

Yep, completely down for me as well. I can understand what Quic must be going through having been in similar situations however this is where maturity of how they run their business comes through.

mrmivec69

68 posts

Master Geek
+1 received by user: 2

#3169383 7-Dec-2023 16:25

I feel bad for the team, this is not what they want obviously and for some of the technical people they have likely not had a lot of sleep!

After a lot of intermittent issues since being with Quic (Since December 22) my wife insisted we change isp as she works from home and was needing to use a 4g hotspot way too often. I convinced her to wait for the network upgrade but after two failed attempts I might be moving on now. I have referred many people to Quic and do hope they succeed, ill check them out again at a later date but for now I need a better quality connection.

Good luck to the team at Quic I know a lot of people want to see you succeed!

mngm

4 posts

Wannabe Geek
+1 received by user: 1

#3169384 7-Dec-2023 16:34

It looks like the rollback completed and they are monitoring.

However, I have about 10 sites just went off again. Anyone experience the same?

Looks like switching between 2 of their pppoe access concentrators.

ratsun81

516 posts

Ultimate Geek
+1 received by user: 183

#3169385 7-Dec-2023 16:39

kennedybaird:

Yeah, totally understand.

For me it's just a challenge as I'm syncing with US, so most of my meetings are early morning.

And today, finally getting into the swing of work and it's randomly dropped out again.

I do feel like it's a little unfair that we're paying full price for such an intermittent service. It's also well and good to say "just change to another provider" - but that ends up being a whole other set of brain cycles

Oh are you paying for a business plan? If so you should hit them up...

If you arent and expecting business level then you really should be looking elsewhere rather than complaining.

1 | 2 | 3 | 4 | 5 | 6 | 7