Packet Loss 03/05/2024

Forums › Vetta and Quic › Packet Loss 03/05/2024

1 | 2 | 3 | 4 | 5 | 6

meow
13581 posts

Uber Geek
+1 received by user: 10914

Moderator

ID Verified

Trusted

Lifetime subscriber

#3227198 6-May-2024 21:00

@MaxineN Yep appears to be the same underlying problem as most others here. I think between you, and others who have provided MTR's it's pointed out the dodgy device that needs attention.

Michael Murphy | https://murfy.nz
Referral Links: Quic Broadband (use R122101E7CV7Q for free setup)

Are you happy with what you get from Geekzone? Please consider supporting us by subscribing.
Opinions are my own and not the views of my employer.

MaxineN

Max
2056 posts

Uber Geek
+1 received by user: 1675

ID Verified

Trusted

Subscriber

#3227200 6-May-2024 21:06

The thing is Michael I'm not feeling the packet loss or seeing it as bad as the rest.

A dropped packet or two is not something I'd cry wolf about but the VPS MTR is absolutely sketch city and I think that's beyond Quic's control (happy to be corrected if they can steer the routing away).

I definitely need to get some of the Microsoft and equinix peers back in my smoke ping so I can see where else (as I do play a lot of valve games and their peer for Sydney is Equinix) this might be cropping up.

I'd rather my results not get used as a sample because I myself deem it inconclusive and I'm not catching it live on the MTRs vs the smoke pings.

20 ICMP pings every 300 seconds is very different to 1 a second.

Again happy to be corrected but to me I'm not finding a fault bar the horrible upstream mess with my VPS.

Ramblings from a mysterious lady who's into tech. Warning I may often create zingers.

ctv

45 posts

Geek
+1 received by user: 8

#3227207 6-May-2024 21:24

MaxineN:

The thing is Michael I'm not feeling the packet loss or seeing it as bad as the rest.

A dropped packet or two is not something I'd cry wolf about but the VPS MTR is absolutely sketch city and I think that's beyond Quic's control (happy to be corrected if they can steer the routing away).

I definitely need to get some of the Microsoft and equinix peers back in my smoke ping so I can see where else (as I do play a lot of valve games and their peer for Sydney is Equinix) this might be cropping up.

I'd rather my results not get used as a sample because I myself deem it inconclusive and I'm not catching it live on the MTRs vs the smoke pings.

20 ICMP pings every 300 seconds is very different to 1 a second.

Again happy to be corrected but to me I'm not finding a fault bar the horrible upstream mess with my VPS.

Yeah, there's quite some heavy loss on the Mercury/Twelve99 route. I've brought this up with Quic but unfortunately, it's out of their control :(

Otherwise, outside of that, everything else looks fine on my end.

For context, here's a destination that is routed through Mercury/Twelve99.

michaelmurfy

meow
13581 posts

Uber Geek
+1 received by user: 10914

Moderator

ID Verified

Trusted

Lifetime subscriber

#3227210 6-May-2024 21:36

ctv: Yeah, there's quite some heavy loss on the Mercury/Twelve99 route. I've brought this up with Quic but unfortunately, it's out of their control :(

Some light reading on this if you're interested: https://www.geekzone.co.nz/forums.asp?forumid=194&topicid=312225&page_no=1#3212167

saf:

I have had a bit of a look for you, and my poke around does hint to add weight to what @Lias has mentioned above.

There are a significant number of submarine cables cut currently between Asia and Europe. With this happening, carriers and providers are being forced to use paths with less capacity, or temporary measures in order to keep packets flowing, which causes congestion on the remaining paths that are working - but hey, moving most packets is better than no packets!

Fixing these cables is proving especially challenging, moreso than normal due to the instability in the region, needing military escorts and defense etc.

Doing some digging, it does look like RETN and/or their provider are striking this problem, which matches both in terms of the traffic path, and explains differences in time of day based on when the working path they're using is congested.

We have already completed some traffic engineering changes a month or so ago in order to route around points of significant congestion due to all the above with the elements we have control over, however we can't fix every path to everywhere, as we can't control the whole path to the remote destination.

eonsim

403 posts

Ultimate Geek
+1 received by user: 192

Trusted

#3227217 6-May-2024 21:59

I've been running MTR for a couple of hours at a 10s interval. Initially I was seeing 2-3x the packet loss from hops 3 and 4 but over the last couple of hours that seems to have evened it's self out

The most noticeable thing remaining is hop 5 is highly inconsistent (as13335.akl.ix.nz), it has ~3x the average latency of the other nodes, and significant variability (standard deviation of the latency is 10x the others). Based in Hamilton but seem to be going directly through Auckland (not sure if that is normal).

Cxxxxxe (xxx.xxx.xxx.xxx) -> 1.1.1.1 (1.1.1.1) 2024-05-06T21:46:51+1200
Keys: Help Display mode Restart statistics Order of fields quit
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. xxxxxxxxxx.lan           0.9% 1282 0.4 0.4 0.3 6.0 0.4
2. bng1-akl1.vetta.net         0.6% 1281 4.6 3.9 3.0 17.8 0.8
3. xe1-2100-146.pe2-akl1.vetta.net 0.8% 1281 4.2 4.3 3.0 66.0 2.0
4. xe1-2100-502.core1-akl2.vetta.net 1.1% 1281 4.0 4.3 3.1 78.8 2.3
5. as13335.akl.ix.nz           1.0% 1281 5.6 13.7 3.9 166.3 22.8
6. one.one.one.one          0.8% 1281 4.8 4.9 3.3 169.4 6.7

squareeyes123

83 posts

Master Geek
+1 received by user: 30

#3228050 8-May-2024 18:28

Still being dogged this. Just got an invoice from Quic and it's salt in the wound. I don't feel great at all being on the backlog when I'm still getting constant micro outages. Has a customer presentation today and it cut right in the middle.
@quic

Quic referral: https://account.quic.nz/refer/415133

New World

Shop on-line at New World now for your groceries (affiliate link).

Delorean

673 posts

Ultimate Geek
+1 received by user: 343

ID Verified

Trusted

Lifetime subscriber

#3228052 8-May-2024 18:34

squareeyes123: Still being dogged this. Just got an invoice from Quic and it's salt in the wound. I don't feel great at all being on the backlog when I'm still getting constant micro outages. Has a Custer presentation today and it cut right in the middle.
@quic

I have recently switch over permanently from One NZ. The connection was very flakey until 5 days ago and now it's rock solid with better performance than One NZ

Maybe it's something still not 100% with your setup?

Referral Link: | Quic Broadband (use R142206E0L2CR for free setup)

quic

236 posts

Master Geek
+1 received by user: 754

Trusted

Quic Broadband

Lifetime subscriber

#3228055 8-May-2024 19:04

Hey team,

I absolutely recognise that I haven't been as active and responsibe in this thread and forums as I should have been and would like to see, which simply comes down to resourcing on our side.

We are a small but mighty team here, however our entire team have been laser focused on recovery (last week) and investigations (this week) to ensure what occoured last week never happens again, which has dominated our time.

(for those who haven't seen, I highly recommend taking a look at the announcement that "bossman" put out last week: https://www.quic.nz/this-weeks-network-outage/)

We do value how important this forum is, and we have completed recruitment for someone who's role will officially include proactive external communication, rather than little ol' me as I can with the (seemingly evaporating) amount of time in my day - so expect better from us here once they're on board and up to speed, I assure you it's an upgrade from me!

In terms of the network side, our investigations (both internal and external) are going well, and we hope to get further information about last week out to you as soon as practical. We have made some minor tweaks in the background overnight this week to iron out any remaining kinks and our network is definitely in a good state currently.

Whilst very early days, we are hearing some reports from customers using Ubiquiti devices (namely UDM's) which seem to be using the WAN DHCP lease with a 60 minute lease time, while we issue 20 minute leases. This causes the device not to renew its lease at the half-way point, and instead expire and get a new lease at expiry, which in turn causes a micro outage. We're not sure if this is a recent update or what at this stage.
We are acquiring one of these for our lab so we can interrogate it further, however if you feel you are impacted by this, feel free to see if you can work it out, or alternatively PPPoE is still always offered to avoid DHCP issues, even temporarily, until we get one of these in our lab to diagnose further.

We truly appreciate each and every one of you here - and while we may not reply to everything while we have our heads down, we do take on board every report, every log, every bit of information we receive to help make what we're doing even better.

Rest assured we'll continue to improve on everything we do - from network to functionality, through to communication here. We're not stopping anytime soon!

🧡

Quic Broadband
quic.nz - The telco who puts you in charge!

squareeyes123

83 posts

Master Geek
+1 received by user: 30

#3228056 8-May-2024 19:16

I'm prepared to entertain the idea it's local out of desperation, but it just seems wrong to me.

Nothing has changed on my network, has been carefully set up and operating perfectly for a couple of years
I have wired devices, and wireless... all experiencing the issue
Others have agreed there is an appliance in my hops that seems to be malfunctioning. Quic themselves seemed to agree, but said they were in a change freeze. FOr some reason this doesn't appear to be a priority.
Others have been experiencing micro drops.
It happens day and night (so not behavior/load related)
Have done a 10min ONT reboot and router reboot
UDM reported drops match my external WAN IP monitoring perfectly

Other info

Setup

Nokia ONT (it did get a FW update I think a few weeks ago)
2K hyperfibre, DHCP, only IPv4. Default MTU, no VLAN
UDM Pro
2 Piholes (this isn't a DNS issue since every device in the house instantly drops at the same time)
~50 devices if you include VMs, IoT. ~20 devices wired
Two Unifi APs - Though I'm testing on ethernet

Things I'll check

Check whether the ONT FW update coincided with the drops
Isolation test / different router
Switch to PPPoE temporarily

Quic referral: https://account.quic.nz/refer/415133

squareeyes123

83 posts

Master Geek
+1 received by user: 30

#3228057 8-May-2024 19:23

@quic

>Whilst very early days, we are hearing some reports from customers using Ubiquiti devices (namely UDM's)

OK this is the smoking gun I need. And I really, really wish I had thought of trying to revert back to PPPoE a week ago. Looks like I had the idea the same time you dropped in. Will report back.

Quic referral: https://account.quic.nz/refer/415133

squareeyes123

83 posts

Master Geek
+1 received by user: 30

#3228058 8-May-2024 19:25

@quic by the way I'm happy to help test if you need

Quic referral: https://account.quic.nz/refer/415133

HP

Shop now for HP laptops and other devices (affiliate link).

michaelmurfy

meow
13581 posts

Uber Geek
+1 received by user: 10914

Moderator

ID Verified

Trusted

Lifetime subscriber

#3228059 8-May-2024 19:29

@squareeyes123 The problem is the micro outages don't seem to be happening to me anymore or to many others and I am not even picking them up myself like I could before - as I've mentioned a number of times now yours does seem to be dropping far more often than anyone else and you're using a dream machine.

Whilst very early days, we are hearing some reports from customers using Ubiquiti devices (namely UDM's) which seem to be using the WAN DHCP lease with a 60 minute lease time, while we issue 20 minute leases. This causes the device not to renew its lease at the half-way point, and instead expire and get a new lease at expiry, which in turn causes a micro outage. We're not sure if this is a recent update or what at this stage.

Could be coincidental perhaps to a firmware update released by Ubiquiti? But perhaps look at logs from your dream machine. Remember, this is a residential service also and dream machines for me anyway have been problematic on other providers too. I'll be interested to know if this can be manually changed.

MaxineN

Max
2056 posts

Uber Geek
+1 received by user: 1675

ID Verified

Trusted

Subscriber

#3228070 8-May-2024 19:35

Just want to confirm so I'm not going mad.

I am pretty sure we're not meant to be seeing packet loss at the BNG.

I have also stood up some other targets more focused on IX and traffic going out of Vetta.

Out of curiosity. @quic xe1-2600-218.pe2-chc4.vetta.net is this port being looked at? Mostly because in most of the MTRs this is where I see the packet loss start. Although below 4% when it happens and over an 8 hour period I saw 0.1% to Cloudflare Auckland from running it this morning before going to work, I am noticing that the route out is where the packet loss starts.

Ramblings from a mysterious lady who's into tech. Warning I may often create zingers.

michaelmurfy

meow
13581 posts

Uber Geek
+1 received by user: 10914

Moderator

ID Verified

Trusted

Lifetime subscriber

#3228080 8-May-2024 20:12

@MaxineN Pretty normal and common practise for network equipment to rate limit ICMP and I know it’s the case with Quic/Vetta also.

MaxineN

Max
2056 posts

Uber Geek
+1 received by user: 1675

ID Verified

Trusted

Subscriber

#3228084 8-May-2024 20:23

Understand that part.

But except from the 2nd of May's small packet loss there were no packets dropped until the above screenshot today from 1:50ish pm

Hence my question of am I going mad?

Ramblings from a mysterious lady who's into tech. Warning I may often create zingers.

1 | 2 | 3 | 4 | 5 | 6