Our new network is here!

Forums › Quic › Our new network is here!

1 | 2 | 3 | 4 | 5 | 6

181 posts

Master Geek

#3186594 25-Jan-2024 23:10

Does anyone see more packet loss? Tried to connect back to work computer (via AWS) and was lagging. Ran a MTR and see packet loss at core1-akl1 node.

Network status point to 40-50% current strain on the chorus - Quic handover link but this is after the BNG into Vetta's network?

Edit:
Dang rose up to 45% now.. is this normal?

michaelmurfy

meow
13217 posts

Uber Geek

Moderator

ID Verified

Trusted

Lifetime subscriber

#3186601 25-Jan-2024 23:58

@prat33k I see some, but far far less than what there was. There is still more work to do as per the OP though. But from your MTR's it appears that pinging a router in the path is seeing some dropped packets which is fine but the end destination doesn't respond to ICMP. Here's one for me to apple.com:

We can see loss on hops 4 and 6 but no loss from the end destination. MTR pings everything in the path but many routers deprioritise ICMP which may show as loss on a MTR. Even right now as I type this I am up to over 500 sent pings and so far only loss on hops 4 and 6 with none to Akamai.

This aligns with what I see elsewhere on monitoring - barely any loss to Apple despite it hopping over to Sydney. I can repeat the test for most destinations and get hardly any loss.

My Smokeping is looking pretty funky currently as I'm testing a few things out and troubleshooting a potential beta bug but a standard Quic connection outside of the Beta looks fine to me. I think there was a slight IPv6 hiccup earlier on but think they're sharing some IPv6 between both the old and new networks currently too so we should see more improvement once their old network is gone.

Also I am pretty sure that network weather chart is showing the old network still on their site which is operating in parallel currently as they move stuff across to the new network.

Michael Murphy | https://murfy.nz
Referral Links: Quic Broadband (use R122101E7CV7Q for free setup)

Are you happy with what you get from Geekzone? Please consider supporting us by subscribing.
Opinions are my own and not the views of my employer.

Gwilson

136 posts

Master Geek

#3186612 26-Jan-2024 06:45

Yes, it certainly looks like things are a bit 'funky' with IPV6 at the moment....

hio77

'That VDSL Cat'
12999 posts

Uber Geek

ID Verified

Trusted

Lizard Networks

Subscriber

#3186613 26-Jan-2024 07:02

The pings being dropped or not responded to in their core is fairly common.

Based on the traces I'd suspect they have gone for a fairly standard MPLS P/PE design, likely with a underlay that's being ran as close to a bgp free core which is becoming the normal these days.

As to a single router along the path showing loss, you can safely ignore this long as the loss does not carry over. In this case what your seeing is simply ICMP deprioritization or rate limiting.

The traces actually look pretty well fine, I would say ipv6 peering does not 1:1 match ipv4 (witch is very common too) cloudflare in particular looks like NZ peering may be down. That could be a maintenance somewhere though too.

#include <std_disclaimer>

Any comments made are personal opinion and do not reflect directly on the position my current or past employers may have.

pc

120 posts

Master Geek

Trusted

Subscriber

#3187074 26-Jan-2024 22:32

bartong: I’m noticing some issue with my Google Home devices (smart speakers) since the upgrade. They seem to be struggling intermittently? Often timing out on responding to commands, music streaming randomly stops, saying the lights or TVs are offline when they clearly are not… just 15 minutes ago I was looking in the Google home app on my phone and everything was just a spinning status.

the 3 ipv6 hosts i was smokepinging are now responding.... and so it seems are my google home devices!

bartong

37 posts

Geek

#3187092 27-Jan-2024 00:39

the 3 ipv6 hosts i was smokepinging are now responding.... and so it seems are my google home devices!

Well well… I’ll keep an eye on mine and see how we go! Mind sharing which servers/IPs you are pinging?

Really could be two things at play then?
1. Quic having some IPv6 routing issues (obviously being actively worked on)
2. Google devices just being totally shit at falling back to IPv4 gracefully if they aren’t getting a response via IPv6…?

michaelmurfy

meow
13217 posts

Uber Geek

Moderator

ID Verified

Trusted

Lifetime subscriber

#3187094 27-Jan-2024 01:53

@pc @bartong That's quite interesting. From my own testing (between beta and a friend on Prod) there is only 1 destination I can't reach and they know about it. Google works fine over IPv6 here so must be related to your connections or the BNG you're on. I've done some hunting around on Smokeping and IPv6 is looking OK currently. Mind letting me know what destinations are failing as I'll ensure I'm monitoring it.

I believe currently there is a bit of a hack going on currently as they're still running 2 networks (their old one, and their new one). I think IPv6 is still a bit of a hack as they can't announce the prefixes from both networks (I remember them talking about this) but it is expected that IPv6 will be way more stable once they've said goodbye to their old network and fully sorted out all their peering.

If you look at the peering list at https://bgp.he.net/country/NZ Vetta were first in NZ for the most BGP peers up until now (for some reason Mercury is showing first tonight which is strange). This would add more things to tidy up I'd say to ensure traffic is going to the right places.

Mighty Ape

Shop now on Mighty Ape (affiliate link).

bartong

37 posts

Geek

#3187247 27-Jan-2024 12:53

@michaelmurfy I have not had the opportunity to set up a smokeping yet, but it does appear my Google speakers are responding with near 100% reliability now so perhaps recent IPv6 routing changes/improvements are making the difference. I’m on bng2, on the Enable network here in Chch.

lucasnz

45 posts

Geek

#3187279 27-Jan-2024 13:51

IPv6 to Google and Facebook returned for me at 2am on Fri morning for me (according to my smokeping).

pc

120 posts

Master Geek

Trusted

Subscriber

#3187310 27-Jan-2024 14:42

ipv6 sites that were down were:

google dns: 2001:4860:4860::8888

youtube: 2404:6800:4006:809::200e

london host from dotcom-monitor : 2a01:4020:0:2e::10

sites that were ok:

cloudflare dns: 2606:4700:4700::1001

other dotcom-monitor sites from california and new york: 2001:470:1:1ab::2 and 2604:eb80:1:6::10

azure: portal.azure.com

my outage was from approx Mon 22nd 18:30 - Fri 26th 01:40

google nest was able to time my cup of tea this morning so all good here!

RunningMan

8912 posts

Uber Geek

#3187806 28-Jan-2024 21:31

michaelmurfy:

[snip] Something else I thought I'd mention is I am using Quic's DNS servers (Primary: 103.250.90.8, Secondary: 202.179.141.8) on my Smokeping instance that can be rather important for some CDN stuff especially Akamai.

Not sure if Apple still use Akamai, but using something other than Quic's DNS has a significant detrimental impact on their software updates. Cloudflare DNS had updates coming through at about 15 Mb/s, far quicker with Quic DNS.

michaelmurfy

meow
13217 posts

Uber Geek

Moderator

ID Verified

Trusted

Lifetime subscriber

#3187827 28-Jan-2024 22:54

@pc Cheers! All those work fine on all my connections and I didn't record any outages on 3 different connections (1 beta, 2 production). I think there was a routing issue but as I said above their IPv6 is a little bit of a hack currently given they're advertising prefixes on 2 different networks. This will improve when they're back to a single network and there is a change for next week to install more route reflectors which should make things a bit better.

Suggestion though - do double-check your IPv6 MTU. Quic should be using the full 1500 byte MTU. I'm going to ping @fe31nz who knows a heck of a load more than me on this subject (and I always look forward to his late night posts).

michaelmurfy

meow
13217 posts

Uber Geek

Moderator

ID Verified

Trusted

Lifetime subscriber

#3187833 28-Jan-2024 23:04

RunningMan: Not sure if Apple still use Akamai, but using something other than Quic's DNS has a significant detrimental impact on their software updates. Cloudflare DNS had updates coming through at about 15 Mb/s, far quicker with Quic DNS.

Apple use a combo of Akamai and ISP caching. Amusingly, their caching hardware is just standard HPe 2U servers filled with SSD's. I'm pretty sure there are caches available over NZIX but yeah it's a bit of a trade-off by not using your ISP's DNS servers most of the times with where you get your content from.

I did find NextDNS to be pretty good at getting somewhat close to your ISP's caches while protecting your privacy though if you want to give that a shot. I did get ~40 Mb/s over that just before doing some Apple TV updates.

RunningMan

8912 posts

Uber Geek

#3187986 29-Jan-2024 09:25

As an example, using Cloudflare DNS you get Apple content from Sydney via Vocus which was hopelessly slow.

bartong

37 posts

Geek

#3187993 29-Jan-2024 09:59

Wow I had not realised that using non-ISP DNS servers could have such an impact on content delivery.

My kids were waiting for an update for Mariokart to download on the Nintendo Switch yesterday and it took nearly an hour, which I thought was ridiculous, so I guess this could explain?

I'm using Unbound recursive DNS resolver (as an upstream for AdGuard Home) so I assume that would suffer the same as using Cloudflare DNS, Quad9, etc since it won't play nice with ISP caching and/or CDNs?

1 | 2 | 3 | 4 | 5 | 6