Recently MTU related issues?

Forums › 2degrees (including Slingshot, Orcon, Flip, Stuff Fibre, MyRepublic, 2talk and Vocus) › Recently MTU related issues?

1 | 2 | 3 | 4 | 5 | 6 | 7 | ... | 8

1232 posts

Uber Geek

#3328499 6-Jan-2025 22:59

yitz:

Also I'm pretty sure fe31nz only advised testing 1508 byte MTU over IPv6 paths but you are testing IPv4 and have said IPv6 is disabled on your LAN.

You should use MTU 1508 for any PPP connection. With IPv6 and not using MTU 1508, IPv6 is broken by the PPP bug that fails to send ICMPv6 "packet too long" replies, so path MTU discovery (mandatory on IPv6) is broken. So long packets get silently dropped and IPv6 will not see the problem. With IPv4, the same bug exists, but it does not actually break IPv4 as the long packets will automatically be fragmented. But fragmenting packets usually means they will be passed from the router offloading hardware to the router CPU for the fragmentation to be done, and router CPUs can rarely cope with handling a full speed 1 Gbit/s stream of full size packets to be fragmented. So your connection speed gets drastically slowed to whatever speed the CPU can handle, typically less than 300 Mbit/s.

Rudster

119 posts

Master Geek

#3328504 6-Jan-2025 23:43

yitz may have hit the issue on the head. I was using the default MSS which appears to have been wrong. Not sure how wrong.

I have updated my config to MTU: 1508, MSS: 1460, ignoring vlan per your comment earlier fe31nz, And everything seems happier.

Not entirely sure that MSS value is correct but 1508 - 8 for ppp, -20 for tcp, -20 for ipv4 is what I got.

Either way, than you both for helping me solve this issue. Now I have more about networking I need to learn...

unexcept

6 posts

Wannabe Geek

#3329074 8-Jan-2025 13:31

Chiming in as someone also running into this issue, and quite confused because nothing seems to work. I can't connect to any github.io sites through my browser.

I'm with 2Degrees, fibre connection with DHCPv4, ipv6 is disabled both on the internet connection and in my local network. I'm running Unifi networking kit, and I have MTU/MSS set to 'auto' (changing this didn't seem to help).

github.io pages work on all non-linux devices on my network, and on both of those linux boxes it's only browser that doesn't work. wget/curl/ping/etc all work.

Running wireshark on one of my network interfaces it looks like the browser TCP handshake is failing because my ACK isn't getting to the other end? Because it looks like the server is sending a retransmisison of the SYN ACK (No. 116).

The packet length for the Client Hello is large, but I feel like that's a red herring (but maybe I'm wrong?), although that could be causing MTU issues? *shrug*

In this wireshark log No. 519 to 590 is the wget call to the same domain and it seems to work fine.

I am at a loss. I've tried a bunch of stuff and nothing has worked, and right now I'm having to work over hotspot from my phone (4g also from 2degrees) just to be able to access open source docs :/

I'm tempted to set MSS to something much lower and restart everything properly to test, but I have tweaked that setting already and it shouldn't need a full restart of everything to take effect.

yitz

2080 posts

Uber Geek

#3329079 8-Jan-2025 13:57

Is it just Github sites you are having trouble with?

Can you try force TLS v1.3 on your browser?

Also I would use a hosts file entry or otherwise to redirect the GitHub hosted site domain from 185.199.109.153 to something like 185.199.109.155 in case just to rule out any issues with the government/DIA filtering on 185.199.109.153.

openmedia

3332 posts

Uber Geek

Trusted

#3329085 8-Jan-2025 14:27

unexcept:

Chiming in as someone also running into this issue, and quite confused because nothing seems to work. I can't connect to any github.io sites through my browser.

I'm with 2Degrees, fibre connection with DHCPv4, ipv6 is disabled both on the internet connection and in my local network. I'm running Unifi networking kit, and I have MTU/MSS set to 'auto' (changing this didn't seem to help).

github.io pages work on all non-linux devices on my network, and on both of those linux boxes it's only browser that doesn't work. wget/curl/ping/etc all work.

I'm on Orcon (2degrees) and seeing similar issues. I also can't access github.io along with a bunch of other sites.

I tried a vanilla Orcon provided router with default settings and had the same issue. Might be worth trying to get our support tickets joined.

Generally known online as OpenMedia, now working for Red Hat APAC as a Technology Evangelist and Portfolio Architect. Still playing with MythTV and digital media on the side.

unexcept

6 posts

Wannabe Geek

#3329094 8-Jan-2025 14:53

yitz:

Is it just Github sites you are having trouble with?

Can you try force TLS v1.3 on your browser?

Also I would use a hosts file entry or otherwise to redirect the GitHub hosted site domain from 185.199.109.153 to something like 185.199.109.155 in case just to rule out any issues with the government/DIA filtering on 185.199.109.153.

It appears to be just github.io hosted sites. And I've been playing with it some more right now and it looks like it's their whole domain range that is the issue as I'm seeing DNS resolve to any of:
- 185.199.108.153
- 185.199.109.153
- 185.199.110.153
- 185.199.111.153

I also don't know what a hosts file entry would really do given that the issue is github.io pages which can include any domain from pages.github.com itself to even something like mindustrygame.github.io

Putting the following in my hosts file:
```
185.199.109.155 mindustrygame.github.io
185.199.109.155 pages.github.com
185.199.109.155 github.io
```
Just gives me "Secure Connection Failed" and going to 185.199.109.155 directly all I see is:
```
Fastly error: unknown domain: 185.199.109.155. Please check that this domain has been added to a service.
Details: cache-akl10326-AKL (185.199.109.155)
```

unexcept

6 posts

Wannabe Geek

#3329097 8-Jan-2025 14:57

No other websites seem to be having issues, and github.com itself works fine! Just pages.github.com hosted sites.

Sharesies

Trade NZ and US shares and funds with Sharesies (affiliate link).

yitz

2080 posts

Uber Geek

#3329098 8-Jan-2025 14:59

unexcept:

Putting the following in my hosts file:
```
185.199.109.155 mindustrygame.github.io
185.199.109.155 pages.github.com
185.199.109.155 github.io
```
Just gives me "Secure Connection Failed" and going to 185.199.109.155 directly all I see is:
```
Fastly error: unknown domain: 185.199.109.155. Please check that this domain has been added to a service.
Details: cache-akl10326-AKL (185.199.109.155)
```

Sorry you're right .155 doesn't work can you try 185.199.109.154

unexcept

6 posts

Wannabe Geek

#3329102 8-Jan-2025 15:08

yitz:

Sorry you're right .155 doesn't work can you try 185.199.109.154

hosts file updated with:
```
185.199.109.154 mindustrygame.github.io
185.199.109.154 pages.github.com
185.199.109.154 github.io
185.199.109.154 beanie-odm.dev
```

the first three work, the fourth fails because of the custom domain and "SSL_ERROR_BAD_CERT_DOMAIN" error.

yitz

2080 posts

Uber Geek

#3329103 8-Jan-2025 15:20

unexcept:

the first three work, the fourth fails because of the custom domain and "SSL_ERROR_BAD_CERT_DOMAIN" error.

Doesn't look like custom domains are serviced on that virtual IP only *.github.io but this was to test possibility of an issue with the DIA filter as packets destined to all the .153 virtual IPs are being routed through it and there have been issues in the past when highly trafficked sites get added. You can remove those lines from the hosts file now after testing, have you tried testing your browser by disabling TLS v1.2 or forcing TLS v1.3?

BTW I can successfully visit all those sites on the .153 IP in a browser TLS version 1.2 (also 2degrees wholesale connection and IP routed through the filter)

unexcept

6 posts

Wannabe Geek

#3329105 8-Jan-2025 15:25

In firefox I tried setting `security.tls.version.min` to 4, but that didn't help. `security.tls.version.max` is by default set to 4 already.

In chrome there isn't a way to force TLS 1.3 as far as I could tell beyond an unclear experimental flag, that also didn't help.

Rudster

119 posts

Master Geek

#3329367 8-Jan-2025 20:08

My issue was 100% mss. Testing mtu to different location, I got varying mtu values with a weird threshold where packets just got lost. Since making the change, all test reach the same mtu as set by the router.

Your symptoms appear to be nearly identical to mine.

yitz

2080 posts

Uber Geek

#3329396 8-Jan-2025 22:11

Just noticed while browsing GitHub pages I'm getting intermittent timeouts now too, DIA filter connection state table might be getting overloaded not taking on new connections. Perhaps it depends on browser/OS networking stack port ranges as to success likelihood? Not noticing any issues with other sites.

Ad block lists are just an example of build artifacts being hosted on github.io so must be pulling lots of traffic. e.g. ublockorigin.github.io

Testing domain provided above:

Anyone know whether 2degrees are subject to the same DDoS as InSPire and might be blocking certain packets/sizes? I think there must be a combination of issues out there...

fe31nz

1232 posts

Uber Geek

#3329399 8-Jan-2025 22:18

@unexcept:

Your Wireshark capture shows something I did not think was possible - packet 109 is shown as having a length of 1949. To do that, it must be made up of two or more fragmented packets that have been defragmented again when Wireshark sees them. Normally, I would expect Wireshark to flag such packets and tell you the packet number(s) of the additional fragments, but that does not seem to be happening in your capture, so maybe the hardware in your Ethernet card automatically defragments packets. As you have noticed, 1949 is too long for a TCP packet on an MTU 1500 connection, so it may be getting dropped as there may not be enough buffer space for Firefox to receive it.

Setting an MSS value seems to fix this problem with https://pages.github.com, and I would expect that it would prevent this sort of long packet. The MSS value is an optional field that can be sent in the SYN packet. It can be a different value for the transmit and receive directions and sets the maximum size of the data part of TCP packets in that direction. You might want to look at the details of your captured SYN packets to see if they do have the MSS field in them and what it is set to. For MTU 1500 packets with the usual IP and TCP headers, the MSS should be 1460. If there are extra headers (eg IPSEC), then it has to be smaller to accomodate those headers. If one of the routers along the path is incapable of handling MSS 1460 packets due to it doing protocol with extra headers (perhaps putting the packets through a GRE tunnel), that could be the cause of the problem here. Such routers are supposed to overwrite the MSS field in SYN packets to the value that they can handle, but that only works if the SYN packets do already have the optional MSS field. And there have been routers and firmware versions that fail to do that overwriting.

insane

3240 posts

Uber Geek

ID Verified

Trusted

#3329401 8-Jan-2025 22:26

Unifi USG + Orcon Fibre (wan1) & 2degrees 4g (wan2), and getting the same issue on the fibre connection to pages.github.com

If I fail the wan over to 4g it starts working fine immediately.

Of course I'm doing very scientific testing using only my mobile phone browser :P

Adjusting MSS on the USG via the UI didn't do anything either.

Might be time to call in @sounddude ...

1 | 2 | 3 | 4 | 5 | 6 | 7 | ... | 8