No internet - 13/09

Forums › Vetta and Quic › No internet - 13/09

1 | 2 | 3

333 posts

Ultimate Geek
+1 received by user: 36

Trusted

#3126876 13-Sep-2023 09:29

Thanks. I got a reply from Shaun from Quic to check it was still broken (it's not)

Would be nice to know what happened, maybe. But yeah, that "not the fibre company, but something is bust" option would be nice :)

Nic Wise - fastchicken.co.nz

breakaway

112 posts

Master Geek
+1 received by user: 5

#3126882 13-Sep-2023 09:41

Mine went down at 9:09 and recovered at approx 9:31. Pings from outside showed TTL expired in transit. Traceroutes showed packets bouncing back and forth between a564073.akl.ix.nz [43.243.21.10] and a564073.auckland.megaport.com [43.243.22.105] until hitting 30 hops at which point they were discarded.

hsvhel

1273 posts

Uber Geek
+1 received by user: 596

ID Verified

#3126884 13-Sep-2023 09:46

They’re very transparent on reporting issues you will come to find, sometimes it takes a little while to get it all written up correctly

Referral Link Quic

Free Setup use R502152EQH6OK on check out

CYaBro

4708 posts

Uber Geek
+1 received by user: 1182

ID Verified

Trusted

#3126888 13-Sep-2023 09:49

Could be a wider issue and not just Quic.
I'm getting notifications that servers are offline yet am still able to access them remotely.
These are not on Quic connections or Chorus fibre.

Opinions are my own and not the views of my employer.

nic.wise

333 posts

Ultimate Geek
+1 received by user: 36

Trusted

#3126892 13-Sep-2023 09:56

hsvhel: They’re very transparent on reporting issues you will come to find, sometimes it takes a little while to get it all written up correctly

assuming thats the case, thats something which keeps me with a provider. AWS is similar - not always THAT quick to update the status page, but their post incident response reports are excellent.

To me it shows that they know what they are doing and can articulate it, which I value a lot.

Nic Wise - fastchicken.co.nz

nic.wise

333 posts

Ultimate Geek
+1 received by user: 36

Trusted

#3126893 13-Sep-2023 09:58

Status page update is up

https://status.quic.nz/issues/6500d885a877246793f76fb3

Nic Wise - fastchicken.co.nz

Dyson

Shop now for Dyson appliances (affiliate link).

jbergler

43 posts

Geek
+1 received by user: 5

ID Verified

#3126898 13-Sep-2023 10:09

Would be great if the status page had a graph (even with no units on the Y axis) of their total edge traffic or something.

Then they don't need to update it but everyone can see theres a wider issue because it fell off a cliff.

quic

236 posts

Master Geek
+1 received by user: 754

Trusted

Quic Broadband

Lifetime subscriber

#3126911 13-Sep-2023 10:18

Hi Team,

Again I can only apologise for the interruption to service today. This is clearly not the experience we want to provide.

Unfortunately this issue began with a backend SQL database providing authentication and traffic accounting locked and was unable to process changes in the database required. While the root cause of this locking is underway, this caused some customers to disconnect and attempt reconnections, which were rejected due a required database needed as part of the authentication and session processes being locked.
When this was resolved, there were backed up transactions to the database which took time to be process and cleared, further delaying resolution for these impacted customers.

I will be making a new thread in this Quic forum later today with some further information on where Quic currently stands, our issues, and what's happening internally to resolve them. I look forward to sharing this with you and will be happy to answer questions once posted.

Quic Broadband
quic.nz - The telco who puts you in charge!

CYaBro

4708 posts

Uber Geek
+1 received by user: 1182

ID Verified

Trusted

#3126912 13-Sep-2023 10:19

Also now getting reports of some sites unable to send emails and EFTPOS terminals offline.

Opinions are my own and not the views of my employer.

GARBAGE

48 posts

Geek
+1 received by user: 13

ID Verified

#3126913 13-Sep-2023 10:24

quic:

Hi Team,

Again I can only apologise for the interruption to service today. This is clearly not the experience we want to provide.

Unfortunately this issue began with a backend SQL database providing authentication and traffic accounting locked and was unable to process changes in the database required. While the root cause of this locking is underway, this caused some customers to disconnect and attempt reconnections, which were rejected due a required database needed as part of the authentication and session processes being locked.
When this was resolved, there were backed up transactions to the database which took time to be process and cleared, further delaying resolution for these impacted customers.

I will be making a new thread in this Quic forum later today with some further information on where Quic currently stands, our issues, and what's happening internally to resolve them. I look forward to sharing this with you and will be happy to answer questions once posted.

Yeah awesome -- I absolutely LOVE this kind of communication.

Thanks!!

jbergler

43 posts

Geek
+1 received by user: 5

ID Verified

#3126921 13-Sep-2023 10:47

quic:

This caused some customers to disconnect and attempt reconnections, which were rejected due a required database needed as part of the authentication and session processes being locked.

I'm outside looking in, so please take this with a grain of salt, but this doesn't line up with the experience I had.
PPP was establishing without issue, my static IP was being assigned as normal and IP connectivity was working to your BNG, just not beyond

$ route -n
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 103.139.184.29 0.0.0.0 UG 0 0 0 pppoe-wan

$ ping 103.250.90.8
PING 103.250.90.8 (103.250.90.8): 56 data bytes
^C
--- 103.250.90.8 ping statistics ---
2 packets transmitted, 0 packets received, 100% packet loss

$ traceroute -n 103.250.90.8
traceroute to 103.250.90.8 (103.250.90.8), 30 hops max, 46 byte packets
1 103.139.184.29 11.072 ms 11.477 ms 11.226 ms
2 *^C

$ ping 103.139.184.29
PING 103.139.184.29 (103.139.184.29): 56 data bytes
64 bytes from 103.139.184.29: seq=0 ttl=64 time=12.306 ms
64 bytes from 103.139.184.29: seq=1 ttl=64 time=12.325 ms

HP

Shop now for HP laptops and other devices (affiliate link).

hio77

'That VDSL Cat'
13036 posts

Uber Geek
+1 received by user: 3896

ID Verified

Trusted

Lizard Networks

Subscriber

#3126923 13-Sep-2023 10:49

quic:

Hi Team,

Again I can only apologise for the interruption to service today. This is clearly not the experience we want to provide.

Unfortunately this issue began with a backend SQL database providing authentication and traffic accounting locked and was unable to process changes in the database required. While the root cause of this locking is underway, this caused some customers to disconnect and attempt reconnections, which were rejected due a required database needed as part of the authentication and session processes being locked.
When this was resolved, there were backed up transactions to the database which took time to be process and cleared, further delaying resolution for these impacted customers.

I will be making a new thread in this Quic forum later today with some further information on where Quic currently stands, our issues, and what's happening internally to resolve them. I look forward to sharing this with you and will be happy to answer questions once posted.

No matter the size of the provider, floods of radius traffic after a fault is fixed is always a secondary painful fault.

Seen the same happen for the largest providers too, it's just generally not externally shared at that size.

props for the open approach to the fault Shaun and team :)

#include <std_disclaimer>

Any comments made are personal opinion and do not reflect directly on the position my current or past employers may have.

michaelmurfy

meow
13580 posts

Uber Geek
+1 received by user: 10912

Moderator

ID Verified

Trusted

Lifetime subscriber

#3126928 13-Sep-2023 10:59

@jbergler Depends here. I saw 2 connections go down but my own connection along with one other stay online. Not sure what their auth process is like but it could put you into a quarantined (or payment required) VLAN for this time blocking off communication to the outside world but still allowing internal communication - this may be what you were seeing but this is at a guess at most.

It is a bit of a pain but Vetta / Quic have had pretty insane growth from what I've seen in a small time but with most of those people being technical it has also likely highlighted additional issues with their network they didn't know about. I do know they've put in a craptonne of work in order to improve their network and I've seen these improvements on my own connection. Having worked for a smaller ISP in the past it is actually rather impressive the work they've done in a rather short time and I do fully appreciate them being really transparent here too.

Michael Murphy | https://murfy.nz
Referral Links: Quic Broadband (use R122101E7CV7Q for free setup)

Are you happy with what you get from Geekzone? Please consider supporting us by subscribing.
Opinions are my own and not the views of my employer.

CYaBro

4708 posts

Uber Geek
+1 received by user: 1182

ID Verified

Trusted

#3126930 13-Sep-2023 11:02

Looks like my users issues are related to a Kordia router so not Quic. :)

Opinions are my own and not the views of my employer.

JinMao

38 posts

Geek
+1 received by user: 9

#3126941 13-Sep-2023 11:20

Hmmmm, seems like I am not alone.

The funny thing is I lost connection to my router in the morning as well.

Thought the problem was with the router. Went to the garage, restarted the router.

Then connected to router, checked PPPoE connections, seemed up. But no Rx. Tried reconnecting PPPoE, then back online.

Might be quic resolved the issue at the same time.

1 | 2 | 3