Packet Loss 03/05/2024

Forums › Vetta and Quic › Packet Loss 03/05/2024

1 | 2 | 3 | 4 | 5 | 6

4708 posts

Uber Geek
+1 received by user: 1182

ID Verified

Trusted

#3228114 8-May-2024 22:13

No smokeping graphs here but I can tell you that before midnight last night the UDM SE shows a spike in latency to 1.1.1.1, up from the 26ms it has been since the big outage to 36ms. Looks like it continued for about 30-60 mins.
Then just after midnight it dropped back to the “normal” 13-14ms and it’s been like that since.

Don’t seem to be having any micro outages here with the UDM SE on dhcp.

Opinions are my own and not the views of my employer.

schmoogol

77 posts

Master Geek
+1 received by user: 16

ID Verified

#3228117 8-May-2024 22:21

CYaBro: No smokeping graphs here but I can tell you that before midnight last night the UDM SE shows a spike in latency to 1.1.1.1, up from the 26ms it has been since the big outage to 36ms. Looks like it continued for about 30-60 mins.
Then just after midnight it dropped back to the “normal” 13-14ms and it’s been like that since.

Don’t seem to be having any micro outages here with the UDM SE on dhcp.

Same for me - looks like the weirdness with CHC-IX is resolved.

Smokeping to 1.1.1.1

Smokeping to 2degrees Christchurch Speedtest

Smokeping to TradeMe

squareeyes123

83 posts

Master Geek
+1 received by user: 30

#3228131 9-May-2024 00:28

Got some help at Unifi. Early Access FW didn't help (was worse - totally killed WAN), nor did some custom FW they supplied. I have left them with logs (and all of my mobile data is smoked).

Finally back online, rolled back to slightly older FW UniFi OS - Dream Machine Pro 3.2.9.

Will monitor for drops @quic and report back.

Note I also tried PPPoE on the EA FW and that didn't help. If I get drops on older FW I'll try PPPoE again.

Quic referral: https://account.quic.nz/refer/415133

squareeyes123

83 posts

Master Geek
+1 received by user: 30

#3228139 9-May-2024 08:02

Still getting drops on 3.2.9 which pre dated my issues.

Tried PPPoE with quic@quic.nz pass quick and getting...

Quic referral: https://account.quic.nz/refer/415133

evilonenz

/dev/urandom
291 posts

Ultimate Geek
+1 received by user: 152

ID Verified

Trusted

Lifetime subscriber

#3228141 9-May-2024 08:05

squareeyes123: Still getting drops on 3.2.9 which pre dated my issues.

Tried PPPoE with quic@quic.nz pass quick and getting...

Remember if swapping from IPoE/DHCP to PPPoE, you will have an outage of up to 22 minutes as Quic won't accept your PPPoE authentication until your DHCP lease has expired.

Smokeping

Referral Links:

Quic - Use code R536299EPGOCN at checkout for free setup
Contact Energy - Use code FRTQDXB for $100 credit

squareeyes123

83 posts

Master Geek
+1 received by user: 30

#3228142 9-May-2024 08:07

Cool it's connected now. I was aware of the downtime and waited for that last night to no avail but might have jumped the gun this morning.

Also have had it connect almost immediately previously. But yeah...

Quic referral: https://account.quic.nz/refer/415133

Mighty Ape

Shop now at Mighty Ape (affiliate link).

evilonenz

/dev/urandom
291 posts

Ultimate Geek
+1 received by user: 152

ID Verified

Trusted

Lifetime subscriber

#3228144 9-May-2024 08:09

squareeyes123:

Cool it's connected now. I was aware of the downtime and waited for that last night to no avail but might have jumped the gun this morning.

Also have had it connect almost immediately previously. But yeah...

It all depends on when your last lease renewal was, so sometimes you'll get lucky, others you won't.

Smokeping

Referral Links:

Quic - Use code R536299EPGOCN at checkout for free setup
Contact Energy - Use code FRTQDXB for $100 credit

squareeyes123

83 posts

Master Geek
+1 received by user: 30

#3228326 9-May-2024 13:25

Just confirming PPPoE has stopped the constant drops. This pretty much rules out all possible causes on my network except the lease policy on the UDM Pro.

While inconclusive it rules a lot out.

I don't have great data on what updates happened when but having rolled back to FW dated prior to the issues last night, I continued to see the drops.

Quic referral: https://account.quic.nz/refer/415133

quic

236 posts

Master Geek
+1 received by user: 754

Trusted

Quic Broadband

Lifetime subscriber

#3228379 9-May-2024 15:11

quic:

Whilst very early days, we are hearing some reports from customers using Ubiquiti devices (namely UDM's) which seem to be using the WAN DHCP lease with a 60 minute lease time, while we issue 20 minute leases. This causes the device not to renew its lease at the half-way point, and instead expire and get a new lease at expiry, which in turn causes a micro outage. We're not sure if this is a recent update or what at this stage.
We are acquiring one of these for our lab so we can interrogate it further, however if you feel you are impacted by this, feel free to see if you can work it out, or alternatively PPPoE is still always offered to avoid DHCP issues, even temporarily, until we get one of these in our lab to diagnose further.

With thanks to the folks here, and our beta testers for the influx of logs from primarily UDM's, we believed this bug (being DHCP not renewing correctly at halftime causing lease to expire and obtain a new lease) is now patched and resolved.

Quic Broadband
quic.nz - The telco who puts you in charge!

squareeyes123

83 posts

Master Geek
+1 received by user: 30

#3228381 9-May-2024 15:16

Some dates for my UDM/ONT Updates and I have to say, nothing really lines up except the Quic P1.

ONT software upgrade for POLTs and related XGSPON ONTs. May 1
UniFi Network has been updated to 8.1.127. April 30, 9:58:44 PM
UniFi Network has been updated to 8.1.113. March 19, 11:22:39 PM
Night 3 ONT Upgrade GEN 3 & GEN 4 March 7 02:52 Completed (25/02/2024 12:00 Event)
UniFi OS has been updated to 3.2.12. February 22, 10:21:34 AM

Looking at my "Internet Restored" alert frequency.

I joined Quic in Jan
18-Feb-2024 I cut over to DHCP
Prior to April I had about 3 drops per month.
12 total in April (starting from the 13th, but definitely an uptick)
The big Quic outage was May 2 ~5pm.
Then from May (3rd) it exploded to 100s

While that complete outage was resolved for me in an hour, the micro drops came thick and fast after that and continue.

Quic referral: https://account.quic.nz/refer/415133

squareeyes123

83 posts

Master Geek
+1 received by user: 30

#3228382 9-May-2024 15:19

quic:

quic:

Whilst very early days, we are hearing some reports from customers using Ubiquiti devices (namely UDM's) which seem to be using the WAN DHCP lease with a 60 minute lease time, while we issue 20 minute leases. This causes the device not to renew its lease at the half-way point, and instead expire and get a new lease at expiry, which in turn causes a micro outage. We're not sure if this is a recent update or what at this stage.
We are acquiring one of these for our lab so we can interrogate it further, however if you feel you are impacted by this, feel free to see if you can work it out, or alternatively PPPoE is still always offered to avoid DHCP issues, even temporarily, until we get one of these in our lab to diagnose further.

With thanks to the folks here, and our beta testers for the influx of logs from primarily UDM's, we believed this bug (being DHCP not renewing correctly at halftime causing lease to expire and obtain a new lease) is now patched and resolved.

Oh great - Where do we get the patch, or was it carrier side?

Quic referral: https://account.quic.nz/refer/415133

AliExpress

Shop now on AliExpress (affiliate link).

quic

236 posts

Master Geek
+1 received by user: 754

Trusted

Quic Broadband

Lifetime subscriber

#3228384 9-May-2024 15:24

squareeyes123:

Oh great - Where do we get the patch, or was it carrier side?

No change your side - patched carrier side for these "fiddly" devices 😉. Should be a bunch happier now!

Quic Broadband
quic.nz - The telco who puts you in charge!

squareeyes123

83 posts

Master Geek
+1 received by user: 30

#3228462 9-May-2024 20:35

Confirmed fixed for me. No drops all day.

Quic referral: https://account.quic.nz/refer/415133

mentalinc

3385 posts

Uber Geek
+1 received by user: 1025

Trusted

#3228465 9-May-2024 20:48

Less than a week to resolve a specific device bug carrier side, after a full scale P1! just wow. well done team Quic!

CPU: AMD 5900x | RAM: GSKILL Trident Z Neo RGB F4-3600C16D-32GTZNC-32-GB | MB: Asus X570-E | GFX: EVGA FTW3 Ultra RTX 3080Ti| Monitor: LG 27GL850-B 2560x1440

Quic: https://account.quic.nz/refer/473833 R473833EQKIBX

squareeyes123

83 posts

Master Geek
+1 received by user: 30

#3228469 9-May-2024 21:07

mentalinc:

Less than a week to resolve a specific device bug carrier side, after a full scale P1! just wow. well done team Quic!

Well there's always two sides to every story...

https://community.ui.com/questions/UDM-Pro-Micro-WAN-Drops-DHCP/03c61ae1-b4ea-43d1-b0c3-aa304caba3f0#answer/b9306dba-d3ee-4cd2-ac16-c7a3bb381192

Quic referral: https://account.quic.nz/refer/415133

1 | 2 | 3 | 4 | 5 | 6