2 Degrees Broadband, ipv6 and UniFi Security Gateway

Forums › 2degrees (including Slingshot, Orcon, Flip, Stuff Fibre, MyRepublic, 2talk and Vocus) › 2 Degrees Broadband, ipv6 and UniFi Security Gateway

1 | 2 | 3 | 4

1228 posts

Uber Geek

#1980218 20-Mar-2018 01:35

vulcannz:

PMTU needed to work on IPv4 as fragmentation breaks many protocols (especially proprietary encrypted stuff). Packets only get fragmented if they get a fragmentation required notification back, PMTU relied on that notification. So if PMTU failed then it's likely packets would not get fragmented and just flat out fail (aka a black hole router).

That is not my understanding at all about how fragmentation works. When an overlong packet hits a router it is too big for, with IPv4 the router just fragments the packet and sends both new packets on. It may or may not send notification back to the source - there is no requirement to do that, but I would hope good routers would do it. A router receiving already fragmented packets does nothing extra - it will not try to join them. When the fragmented packets get to their destination, they are buffered until both (or all) the fragments are available and then they are put back together again.

If the source of overlong packets gets a notification about it, it can choose to reduce the MTU it is using, but again, in IPv4, that is not required, as the packets will continue to be fragmented at any router where that needs to happen.

Fragmentation is a big burden on a router. If a packet can just be passed through intact, often that is done with acceleration hardware and the router CPU never sees the packet. But when it needs fragmentation, the hardware normally has to hand off the packet to the CPU, which will have to build two packets from that one packet and send them on. That costs a fair bit of CPU effort, and also needs twice as much buffer space plus a bit more for the second set of headers.

A black hole router is either so heavily overloaded that it must drop lots of packets, or is broken and needs to be fixed. When a packet needing fragmentation hits a router that is heavily overloaded, it may well be a good candidate for that router to drop, as the CPU burden of doing the fragmentation will likely increase the overloading, and the extra buffer space needed almost certainly will increase the overloading. And the router may well be so overloaded that it will also drop any attempt to send back notification that it dropped the packet. This sort of overloading can happen because of a reduced MTU causing most larger IPv4 packets to need fragmentation, and is one of the major reasons IPv6 does not allow fragmentation and requires that PMTU works. PMTU was added to IPv4 to help prevent overloading of routers, but as it is not required to work on all IPv4 devices, it often does not succeed due to one or more devices along the path not supporting it. When it does work, it is excellent.

So, fragmentation does not break any IPv4 protocols. Enough packet loss (caused by a black hole router or any other reason) can break all IPv4 protocols, but some protocols take much more packet loss before they break. Encrypted protocols are particularly sensitive to packet loss unless they have a mechanism that allows them to resend just an identical copy of lost packets. If losing a packet requires a complete resend of the entire message in order for the encryption to work, protocols like that will break easily with minimal packet loss.

kiwibob1

109 posts

Master Geek

ID Verified

Lifetime subscriber

#2004208 28-Apr-2018 18:20

michaelmurfy:

I, Too am running IPv6 at a MTU of 1500 and 1508 on the physical port.

Following the suggestions in this thread I've changed to using the IPv6 support in the USG (5.7.23) and noticed that if the MTUs are changed to these values rather than the defaults then the IPv6 tests here (http://test-ipv6.com/index.html.en_US) run significantly faster.

vulcannz

436 posts

Ultimate Geek
Inactive user

#2004856 30-Apr-2018 09:22

fe31nz:

vulcannz:

PMTU needed to work on IPv4 as fragmentation breaks many protocols (especially proprietary encrypted stuff). Packets only get fragmented if they get a fragmentation required notification back, PMTU relied on that notification. So if PMTU failed then it's likely packets would not get fragmented and just flat out fail (aka a black hole router).

That is not my understanding at all about how fragmentation works. When an overlong packet hits a router it is too big for, with IPv4 the router just fragments the packet and sends both new packets on. It may or may not send notification back to the source - there is no requirement to do that, but I would hope good routers would do it. A router receiving already fragmented packets does nothing extra - it will not try to join them. When the fragmented packets get to their destination, they are buffered until both (or all) the fragments are available and then they are put back together again.

It depends, on edge equipment the common configuration is for the edge router to fragment the packet. As step away from the edge the telco routers will drop packets and should send back an ICMP message fragmentation required.

Fragmenting can break a lot of applications so telcos tend to stay away from it. Your statement about fragmentation breaking no IPv4 protocols is quite incorrect, off the bat it will break IPSEC (and of course TLS, and SSL). Anything with encryption does not like the packet being fiddled with.

I see a lot of performance and applications woes on edge equipment where fragmentation is required. Avoiding fragmentation is a good thing.

fe31nz

1228 posts

Uber Geek

#2005564 1-May-2018 00:07

I have no practial experience using IPSEC. But everything I have read about it says that it should be able to be fragmented without problems. Fragmentation means that the original packet gets put back together at the destination - so the encryption is unaffected as the full encrypted packet is available to be decrypted. But if the connection is affected by problems with dropped packets triggered by MTU problems, then IPSEC will have performance issues with fragmentation. And some IPSEC implementations seem to want to set the DF = Do not Fragment bit, which means they will not work if there are MTU issues. That is likely done because of the performance issues with encryption when there is fragmentation happening - fragmentation is often associated with overloaded routers and packet loss, because the router doing fragmentation causes it to become overloaded.

TLS and SSL certainly are unaffected by fragmentation - as long as all the fragments of the packet arrive at the other end, there is no problem. If they were affected by fragmentation, there would be huge problems with TLS and SSL connections around the Internet that just do not happen. So I wonder where you are getting the idea that fragmentation kills encryption. Fragmentation is done on fully formed packets with all the headers and checksums complete. The fragments get put back together again at the destination in such a way that the original packet with all its headers and checksums is identical to the packet that got fragmented. Hence any encryption inside the packet is completely unaffected - as long as all the fragments arrive.

vulcannz

436 posts

Ultimate Geek
Inactive user

#2006277 1-May-2018 21:11

I have lots of experience with IPSEC, TLS, SSL and other proprietary encrypted protocols and can assure you fragmentation is a huge issue.

You have two major issues, for IPSEC to handle fragmentation properly PMTU Discovery needs to work - and everyone knows PMTUD is fundamentally broken. 2nd IPSEC is intentionally very sensitive to packet changes, it just doesn't like fragmentation - it's also not "pure" - it is normally encapsulated within a UDP packet for NAT traversal and other network oddities (for many years home/SMB routers did not support AH/ESP protocol routing). It is very common to see VPNs breaking simply because an MTU is set too high. Signs of MTU issues are poor performance, VPN dropouts, and session disconnects within VPNs (very common to show up when RDP is used).

For TLS/SSL you have the same PMTUD challenge. Signs of MTU issues here show up as half loaded secure web sites (especially image heavy), slow internet performance, and poor SSL VPN experiences.

I've also seen lots of issues with proprietary encrypted protocols like Skype and Aspera. In my last job I was doing beta testing for a few products, one in particular was a cloud service using uFTP. The devs had set an MTU of 1500 that was causing havoc. Initially this was an Asia Pac issue - but it was also seen in Europe and some US states. They'd never come across it before. I guess in NZ we experience more because we typically have more hops to travel and are more likely to hit crappy or overwhelmed routers.

As I've said before, my field is firewalls and remote access (IPSEC/SSL VPN etc), and it's been my focus since 2002. If I had a dollar for every time I had sort out a VPN/MTU isssue I'd have a few grand :)

fe31nz

1228 posts

Uber Geek

#2006334 2-May-2018 00:29

Yes, there are real problems getting encrypted connections from one side of the Internet to the other. But the problem is not fragmentation as such, it is lost packets. If all the fragments of an encrypted packet reach the destination, nothing gets broken. Fragmentation does NOT change the contents of the reassembled packet from what was in the packet that was originally sent. Encrypted connections are inherently much more sensitive to packet loss, that is the problem you are seeing. When the MTU changes to a smaller size, it often cause packet loss by overloading the router as it tries to handle all the fragmenation. So the loss of one fragment of one packet then causes the encrypted connection to have to resend that packet, or if it is a bad encrypted protocol, it might have to back up and resend lots of packets to re-establish the encryption. If the connection is SSL, and the underlying TCP connection has negotiated the option of just resending one lost packet, then it is relatively unaffected by packet loss, when the loss is not too frequent. But that is the problem with overloaded routers - loss is not infrequent if you keep sending long packets that need fragmentation, as those are the packets most likely to be dropped by an overloaded router. If you send unencrypted connections through an MTU change and the router is overloaded, they too will suffer badly if they keep on sending long packets. But fairly large packet loss can be coped with as an unecrypted connection does not reach a point of needing to re-establish the encryption. You can get a working unecrypted connection with over 50% packet loss. It will be very slow, but the connection will not necessarily drop.

vulcannz

436 posts

Ultimate Geek
Inactive user

#2006367 2-May-2018 07:53

Well all I can say is MTU can have a work/no work impact on encrypted protocols. In theory you are correct, but in practicality some things just don't work as they should (like PMTUD).

Equinox IT

Cloud spending continues to surge globally, but most organisations haven’t made the changes necessary to maximise the value and cost-efficiency benefits of their cloud investments. Download the whitepaper From Overspend to Advantage now.

TheoM

228 posts

Master Geek

ID Verified

Trusted

#2008941 6-May-2018 22:47

Well, here I am a bit later. It works (although I'm now on Vodafone HFC, so I don't need the additional config). It's good to be in the IPv6 lane!

Hi! I'm TheoM, but you know that already. I run Linux mirrors in NZ together with 2degrees. Like a mirror added? PM me!

https://theom.co.nz | https://theom.nz | https://mirrorlist.mirrors.theom.nz | Providing Free Mirrors Since Ages Ago™

sorceror

163 posts

Master Geek

#2009269 7-May-2018 14:49

fe31nz:

Yes, there are real problems getting encrypted connections from one side of the Internet to the other. But the problem is not fragmentation as such, it is lost packets. If all the fragments of an encrypted packet reach the destination, nothing gets broken. Fragmentation does NOT change the contents of the reassembled packet from what was in the packet that was originally sent. Encrypted connections are inherently much more sensitive to packet loss, that is the problem you are seeing. When the MTU changes to a smaller size, it often cause packet loss by overloading the router as it tries to handle all the fragmenation. So the loss of one fragment of one packet then causes the encrypted connection to have to resend that packet, or if it is a bad encrypted protocol, it might have to back up and resend lots of packets to re-establish the encryption. If the connection is SSL, and the underlying TCP connection has negotiated the option of just resending one lost packet, then it is relatively unaffected by packet loss, when the loss is not too frequent. But that is the problem with overloaded routers - loss is not infrequent if you keep sending long packets that need fragmentation, as those are the packets most likely to be dropped by an overloaded router. If you send unencrypted connections through an MTU change and the router is overloaded, they too will suffer badly if they keep on sending long packets. But fairly large packet loss can be coped with as an unecrypted connection does not reach a point of needing to re-establish the encryption. You can get a working unecrypted connection with over 50% packet loss. It will be very slow, but the connection will not necessarily drop.

you are right that the problem is lost packets, but the root cause IS due to fragmentation (rather the lack of).

instead of just sending oversized packets and expecting them to be fragmented, some endpoints try and go through the PMTUD process by sending large packets with the DF bit set. when these packets die the death a few hops upstream and the sender doesn't get an ICMP reply for whatever reason (firewall or w/e), the sender *should* back off and either send smaller packets or stop setting DF, but this doesn't always happen. so you get these weird half working 'black hole' connections (although to be fair this was more of a problem in the past).

there is nothing in this process that's specific to VPNs etc, it's just more likely to happen with them as the packets are larger due to overheads.

Noodles

487 posts

Ultimate Geek

#2067230 3-Aug-2018 08:21

Thanks to all the help here I had ipv6 working on our USG with unifi controller 5.7. Upon updating to 5.8 this seems to have broken. This is the error message I get:

configuration commit error. Error message: { "DELETE" : { "failure" : "0" , "success" : "1"} , "SESSION_ID" : "b2a2210e11b2bc450e3d48b06d" , "SET" : { "error" : { "interfaces ethernet eth2 vif 10 pppoe 2 dhcpv6-pd pd 0 interface eth0 prefix-id ..." : "invalid prefix ID\n\nValue validation failed\n"} , "failure" : "1" , "success" : "1"}}

The GUI won't let me remove the prefix ID from the LAN interface. Any suggestions on getting this working again?

ANglEAUT

2320 posts

Uber Geek

Trusted

Lifetime subscriber

#2067384 3-Aug-2018 11:29

Noodles: ... I had ipv6 working on our USG with unifi controller 5.7. Upon updating to 5.8 this seems to have broken. This is the error message I get:

configuration commit error. Error message: { "DELETE" : { "failure" : "0" , "success" : "1"} , "SESSION_ID" : "b2a2210e11b2bc450e3d48b06d" , "SET" : { "error" : { "interfaces ethernet eth2 vif 10 pppoe 2 dhcpv6-pd pd 0 interface eth0 prefix-id ..." : "invalid prefix ID\n\nValue validation failed\n"} , "failure" : "1" , "success" : "1"}} ...

Welcome to my world. Oh, and Benoire complained about losing IPv6 as well.

Please keep this GZ community vibrant by contributing in a constructive & respectful manner.

Noodles

487 posts

Ultimate Geek

#2067385 3-Aug-2018 11:34

Well, I'm glad I'm not alone, but it doesn't look like there's a solve yet (other than downgrading to 5.7.x). I tried contact Ubiquiti support, but because ipv6 is "alpha" they don't offer any support. Because ipv6 is really new /s

Benoire

2798 posts

Uber Geek

#2067423 3-Aug-2018 12:27

If you have only one network, then do not assign a prefix id and it will work fine. If you have more than one network, again don't assign a prefix ID but only enable it on one network... it will work fine, otherwise you get the error above. Controller version 5.8.x is broken completely for multiple networks on IPv6. Not sure if UBNT have acknowledged this yet but enabling one network only with no prefix is the only way to get IPv6 running.

Noodles

487 posts

Ultimate Geek

#2067432 3-Aug-2018 12:37

I have LAN and WAN networks setup on the controller. The GUI stops me from removing the prefix ID (errors when I save). I'm sure it's possible to override this with the config.json or whatever, but not sure what I'm doing.

Benoire

2798 posts

Uber Geek

#2067435 3-Aug-2018 12:45

Disable IPv6 at the WAN level first I believe, then delete the prefix's and then enable again... I think that should fix it, effectively it appears a problem with the WAN IPv6 enabled as you can keep the prefixs and IPv6 network settings on with wan disabled and it works ok.

1 | 2 | 3 | 4