Packet loss to some servers

Forums › Quic › Packet loss to some servers

Behodar

10435 posts

Uber Geek

Trusted

Lifetime subscriber

#318584 31-Jan-2025 22:17

In the middle of a game so this is quick and dirty, but is anyone else getting dropped packets to e.g. 1.1.1.1 or www.geekzone.co.nz? Pinging the BNG seems fine.

Can supply details later.

Edit: Now that I'm dead...

BNG's all fine:

PING 103.243.102.38 (103.243.102.38): 56 data bytes
64 bytes from 103.243.102.38: icmp_seq=0 ttl=63 time=11.183 ms
64 bytes from 103.243.102.38: icmp_seq=1 ttl=63 time=10.409 ms
64 bytes from 103.243.102.38: icmp_seq=2 ttl=63 time=11.065 ms
64 bytes from 103.243.102.38: icmp_seq=3 ttl=63 time=10.571 ms
64 bytes from 103.243.102.38: icmp_seq=4 ttl=63 time=10.856 ms
64 bytes from 103.243.102.38: icmp_seq=5 ttl=63 time=11.102 ms
64 bytes from 103.243.102.38: icmp_seq=6 ttl=63 time=10.369 ms
64 bytes from 103.243.102.38: icmp_seq=7 ttl=63 time=10.366 ms
64 bytes from 103.243.102.38: icmp_seq=8 ttl=63 time=10.546 ms
64 bytes from 103.243.102.38: icmp_seq=9 ttl=63 time=11.411 ms
64 bytes from 103.243.102.38: icmp_seq=10 ttl=63 time=10.705 ms
^C
--- 103.243.102.38 ping statistics ---
11 packets transmitted, 11 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 10.366/10.780/11.411/0.348 ms

Geekzone isn't:

PING www.geekzone.co.nz (172.67.74.108): 56 data bytes
64 bytes from 172.67.74.108: icmp_seq=0 ttl=59 time=17.680 ms
64 bytes from 172.67.74.108: icmp_seq=1 ttl=59 time=17.755 ms
64 bytes from 172.67.74.108: icmp_seq=2 ttl=59 time=17.046 ms
Request timeout for icmp_seq 3
64 bytes from 172.67.74.108: icmp_seq=4 ttl=59 time=15.653 ms
64 bytes from 172.67.74.108: icmp_seq=5 ttl=59 time=23.232 ms
64 bytes from 172.67.74.108: icmp_seq=6 ttl=59 time=10.438 ms
64 bytes from 172.67.74.108: icmp_seq=7 ttl=59 time=10.556 ms
64 bytes from 172.67.74.108: icmp_seq=8 ttl=59 time=10.366 ms
^C
--- www.geekzone.co.nz ping statistics ---
9 packets transmitted, 8 packets received, 11.1% packet loss
round-trip min/avg/max/stddev = 10.366/15.341/23.232/4.305 ms

And neither is 1.1.1.1:

PING 1.1.1.1 (1.1.1.1): 56 data bytes
64 bytes from 1.1.1.1: icmp_seq=0 ttl=60 time=17.777 ms
Request timeout for icmp_seq 1
Request timeout for icmp_seq 2
64 bytes from 1.1.1.1: icmp_seq=3 ttl=60 time=17.770 ms
Request timeout for icmp_seq 4
64 bytes from 1.1.1.1: icmp_seq=5 ttl=60 time=16.588 ms
64 bytes from 1.1.1.1: icmp_seq=6 ttl=60 time=17.582 ms
^C
--- 1.1.1.1 ping statistics ---
8 packets transmitted, 4 packets received, 50.0% packet loss
round-trip min/avg/max/stddev = 16.588/17.429/17.777/0.492 ms

As you can see, I'm on BNG1-AKL2. My connection is in Whakatane via Chorus.

Edit: Was this maintenance supposed to help? ("currently introducing sporadic latency and loss spikes to certain internet destinations"). If so, it seems that it didn't work!

MaxineN

Max
1761 posts

Uber Geek

ID Verified

Trusted

Subscriber

#3337937 31-Jan-2025 23:31

If you haven't joined the discord then I'll paraphrase.

On the 23rd there was a maintenance to upgrade firmware that didn't go as planned and brought on 3 issues (was never picked up in the lab due to not enough load, maybe they need more beta testers)

On top of that there were 2 more issues introduced in the same week on top of trying to troubleshoot the 3 issues that's been acknowledged by the vendor.

There is going to be a PIR, but as a gamer and as someone who does a lot of WFH and real time content... This definitely hurt and I am very reminiscent of what I said in the two posts linked here.

My confidence is dwindling. Hoping that they fix it for good... But yeah... That hurt...

Ramblings from a mysterious lady who's into tech. Warning I may often create zingers.

I use quic. They're a great ISP for techies. Use my referral I get money off my bill! You can get the setup fee waived too via this code when you order! R177510EBNVXP

lloydw

21 posts

Geek

#3337938 31-Jan-2025 23:33

The team has had a rough week, vendor firmware upgrades that introduced new bugs and internal hardware failures. Believe they had to rollback changes last night (emergency maintenance) to get to a more stable state again. They’re taking a break and regrouping next week.

Join the official customer discord via the portal to get more in depth (and live) updates.

michaelmurfy

meow
13218 posts

Uber Geek

Moderator

ID Verified

Trusted

Lifetime subscriber

#3338077 1-Feb-2025 16:03

As others said join in the fun on Discord. You can get to this from your Quic account. Often more is posted there as it's a "only customers" forum so there are more technical information also there I don't quite want to post here.

I did however get permission to crosspost this here. The Quic guys have had a hell of a week and have been resting but this was just posted in their Discord:

Hi all, we know there are still a couple of gremlins kicking around.
We believe these are now isolated to two issues, one being an appliance network interface or card, and another being a switch with odd behaviour compared with its peers.
The former has been worked on with traffic reduced through this appliance, and plans developing for the latter.
We hear and understand your frustration, however after this week there is also a human fatigue factor at play, where completing works under fatigue is a bad idea.
As I say, plans in place, and we’ll keep you updated as always with progress as plans are firmed up and changes planned.
Appreciate your patience and understanding. PIR will be released and available once all back to 100%.

This has been caused by a few separate bugs with their hardware vendor along with equipment failure. Their network has actually done pretty well as I have not seen a full outage, but there's no hiding there has been packet loss.

They are still working on things behind the scenes and many of us have provided them valuable information for troubleshooting that they're working through. But rest for their engineers while the network is in more of a stable state is more important than working on this issue 24/7 as tired engineers means silly mistakes / more issues.

Hopefully we'll see this fully fixed up next week. I'm sure they'll post here too.

Michael Murphy | https://murfy.nz
Referral Links: Quic Broadband (use R122101E7CV7Q for free setup)

Are you happy with what you get from Geekzone? Please consider supporting us by subscribing.
Opinions are my own and not the views of my employer.

Behodar

10435 posts

Uber Geek

Trusted

Lifetime subscriber

#3338079 1-Feb-2025 16:10

Sorry, I thought I'd responded to this earlier. I did begrudgingly join Discord (not a fan of it) so I now have access to the information. Hopefully it's on its way to being sorted.

I appreciate tiredness, but you can hopefully also see my frustration in having the status page saying "operating normally" when they have known issues.

michaelmurfy

meow
13218 posts

Uber Geek

Moderator

ID Verified

Trusted

Lifetime subscriber

#3338081 1-Feb-2025 16:18

Behodar: I appreciate tiredness, but you can hopefully also see my frustration in having the status page saying "operating normally" when they have known issues.

Agreed. I did raise this. Something will be popped on their status page once a plan is confirmed.

Lias

5581 posts

Uber Geek

ID Verified

Trusted

Lifetime subscriber

#3338082 1-Feb-2025 16:30

As others have noted their communication via Discord has been excellent. Quic are far from the only ISP to have suffered degraded performance due to things issues largely outside their control, including far larger and better resourced ISPs and I think sometimes we're a bit quick to forget that. IMHO Quic took some very good learnings from the teething trouble a year or two ago and are now exceptionally good at acknowledging when things go pear shaped. Frustrating as the packet loss is as a gamer and a full time remote worker, I'm not planning on going anywhere.

I'm a geek, a gamer, a dad, a Quic user, and an IT Professional. I have a full rack home lab, size 15 feet, an epic beard and Asperger's. I'm a bit of a Cypherpunk, who believes information wants to be free and the Net interprets censorship as damage and routes around it. If you use my Quic signup you can also use the code R570394EKGIZ8 for free setup.

Lias

5581 posts

Uber Geek

ID Verified

Trusted

Lifetime subscriber

#3338083 1-Feb-2025 16:31

Behodar:

Sorry, I thought I'd responded to this earlier. I did begrudgingly join Discord (not a fan of it) so I now have access to the information. Hopefully it's on its way to being sorted.

I appreciate tiredness, but you can hopefully also see my frustration in having the status page saying "operating normally" when they have known issues.

I think that's quite reasonable feedback, those of us who live on Discord perhaps forget that not everyone does :-P

Backblaze

Backblaze Unlimited Backup. World’s easiest cloud backup. Get peace of mind knowing your files are backed up securely in the cloud (affiliate link).

michaelmurfy

meow
13218 posts

Uber Geek

Moderator

ID Verified

Trusted

Lifetime subscriber

#3338158 2-Feb-2025 01:25

Some maintenance was done tonight to isolate a core switch from some services hopefully improving things: https://status.quic.nz/issues/679da7bd386fb3241a1f08a9

Now in the monitoring stage, it's early but things are looking far better from what I can see from their work today. There will still potentially be some packet loss to some other services but should not be anywhere as bad (hopefully).

RunningMan

8913 posts

Uber Geek

#3338180 2-Feb-2025 09:28

Behodar:Sorry, I thought I'd responded to this earlier. I did begrudgingly join Discord (not a fan of it) so I now have access to the information. Hopefully it's on its way to being sorted.

I appreciate tiredness, but you can hopefully also see my frustration in having the status page saying "operating normally" when they have known issues.

Whilst this is going a little OT, I think the point about info only on Discord shouldn't be overlooked.

Firstly, don't get me wrong, I think it's awesome for a company to engage with it's customers and provide them the level of detail that's over on Discord. However, as @Behodar points out, not everyone uses or has access to that platform for a multitude of reasons, so having a brief note in the status page which anyone can see is pretty important - knowing the issue is outside your own network, rather than chasing something internal that doesn't exist is fairly essential info for the customer to have.

I liken it to companies that require you to use Facebook to contact them. Not everyone wants to (nor should have to) create an account with whatever 3rd party platform just to get basic info from a company they already have a relationship with.

Again, not bagging @Quic for providing info on Discord (it's great), nor that they've had an issue (it happens), but this has come up in feedback from previous incidents regarding status updates. A single sentence saying we've got a packet loss issue, it affects (say) this geographical area, and we're working on it is all that's needed to inform the majority of customers. For those that need more in-depth info, it's there on Discord.

michaelmurfy

meow
13218 posts

Uber Geek

Moderator

ID Verified

Trusted

Lifetime subscriber

#3338202 2-Feb-2025 12:26

RunningMan: Whilst this is going a little OT, I think the point about info only on Discord shouldn't be overlooked.

Simple answer to that is it is a self service ISP. They do also publish status page updates when appropriate. But, having a community you can join where it can be restricted to customers only (via a bot) is a great idea and some staff members choose to engage with customers there too.

It's a feature-add over anything else. But many communities use Discord these days so think it's a great idea considering many people are already on it. But no matter what you do, not everyone is going to be happy.

At-least people were able to provide some updates outside of their Discord walled garden still and affected customers were informed unlike the recent example we saw recently with 2degrees having major packet loss to multiple services: https://www.geekzone.co.nz/forums.asp?forumid=81&topicid=318294 where there were zero updates to any status page, tickets being closed without action on a full support ISP.

RunningMan

8913 posts

Uber Geek

#3338221 2-Feb-2025 13:52

michaelmurfy:[snip] Simple answer to that is it is a self service ISP. They do also publish status page updates when appropriate.

Yep, that's exactly my point. Because it's self service you need a single place you can go to check whether or not there is a problem they are aware of (i.e. the status page). You can't service yourself if you can't make a quick check to see if there's a known issue upstream of you.

Given the number of customers just on here that it seems to have impacted, it would have been appropriate for a status page update, but this wasn't done until some time down the track.

Look at it from a customer retention viewpoint. Selling internet connections is a fickle business, and it's really easy for customers to leave pretty much instantly if they get annoyed. Quic are a company with a lot of automated systems in order to facilitate it being self service - if it isn't already, then one of those automated systems should be a quick, easy way for a brief status update to be published.

Behodar

10435 posts

Uber Geek

Trusted

Lifetime subscriber

#3339399 5-Feb-2025 08:20

For what it's worth, I've yet to see the problem since the Sunday morning maintenance :)