Optimised EdgeRouter X Configuration for Spark UFB PPPoE Internet Connection

Forums › LAN (ethernet/Wifi/routers/Bluetooth) › Optimised EdgeRouter X Configuration for Spark UFB PPPoE Internet Connection

1 | 2

fe31nz

1232 posts

Uber Geek

#2822182 1-Dec-2021 01:42

The 1508 MTU only goes on the WAN Ethernet interface. What is going on here is that the PPPoE protocol has an extra 8 bytes of headers. So when you send a full size ordinary Ethernet packet (MTU 1500) over a PPPoE connection, either the packet has to be spilt into smaller packets (which happens automatically for IPv4 but slows the connection), or the packet will be dropped (which is what happens with IPv6). So either you need to send smaller packets (by setting a smaller MTU of 1492 to allow for the extra 8 bytes), or you need to increase the size of packets on the PPPoE connection by 8 bytes to allow 1500+8 byte packets to be sent. This is a known problem, and the fibre companies allow for it by overprovisioning their fibre connections by 8 bytes. But the default settings on most routers will only be set up for an MTU of 1500 on the PPPoE connection. And old versions of PPPoE software did not support MTUs larger than 1500 either. Fortunately, the versions of PPPoE software around now do support larger MTUs, and good routers also support larger MTUs (at least up to 1508). So you configure your router's WAN Ethernet port to MTU 1508, and then configure the VLAN 10 connection over that port also to MTU 1508, and configure the PPPoE MTU to 1500, which means the actual packet size of the PPPoE packets after conversion to PPPoE protocol will be 1508. Packets arriving over the PPPoE connection will be stripped of the PPPoE headers and converted to standard Ethernet packets with MTU 1500. So the rest of your network (ordinary Ethernet and WiFi connections) will use MTU 1500.

An extra complication comes if you are using IPv6. IPv6 does not allow automatic fragmentation of packets, so if a 1500 byte packet gets sent over a PPPoE connection with the default PPPoE MTU of 1492, it will be dropped. The IPv6 specifications say that if a too long packet is ever dropped, the interface dropping it MUST send an ICMPv6 packet back to the transmitter saying that packet was dropped and giving the reason as "too long". But there is a bug in all PPPoE implementations I have met that the PPPoE software never sends ICMPv6 "too long" packets, in violation of the IPv6 specifications. This breaks IPv6 - the long packets just disappear and there is no mechanism in IPv6 for discovering this or correcting the problem. People usually find this problem by Facebook stopping working - Facebook is IPv6 enabled, and its front page sends full length 1500 byte IPv6 packets, so if you enable IPv6 on a network without a fix for the PPPoE problem, Facebook and all similar IPv6 enabled web sites stop working properly. So if you are using IPv6 over a PPPoE connection with MTU less than 1500, you have set a smaller IPv6 MTU. The way this is done is to have the router broadcast the smaller MTU in its Router Advertisement IPv6 packets. You normally have to set this up specially, and not all routers that do IPv6 have a way to do this. And the RA packet MTU setting will be used for all IPv6 traffic on your network, not just packets heading out the PPPoE connection, so this makes IPv6 less efficient throughout your entire network. It is much better to increase the MTU of the PPPoE packets to 1500 if your router supports that. And it makes the IPv4 large packets work better too.

AndyT

163 posts

Master Geek

#2822753 1-Dec-2021 21:32

Hi fe31nz.

Thank you for this very comprehensive information .... I'm learning fast!

I plugged in the new MTU values earlier this evening and speeds did improve approx. 10% from about 592Mbps DL / 533Mbps UL to about 645Mbps DL / 533Mbps UL and reached 685Mbps DL at best but UL dropped to 381Mbps at the same time. This was at about 5.30pm to 6.00pm this evening i.e. peak'ish time, so I'll, re-test at a non-peak time and see if speeds are then better.

Out of interest, would the same principles apply and improve speeds at our rural place where it's Spark rural 4G wireless broadband with a Huawei B681 modem with all functionality switched off other than being the basic modem, feeding from a DMZ into an ER-X? The B618 settings has VLAN10, PPPoE MTU at default 1492, Dynamic IP MTU at default 1500. All ER-X ethernet interfaces are at MTU 1500, although TPC MSS Clamping is on at MTU 1452. Should I change the PPPoE to 1500, Dynamic IP to 1508, leave all ethernet interfaces at 1500 and turn off TCP MSS Clamping?

fe31nz

1232 posts

Uber Geek

#2822842 2-Dec-2021 01:16

I know nothing about Spark 4G broadband. Using MTU 1500/1508 will only work if they have allowed for it when setting up their network. It should not hurt to try those settings, but I suspect that if 1508 was available, they would have sent you the 4G router set up that way.

You should also spend some time trying different speed testing methods. That is usually where the problems lie, rather than in the local network or in your ISP's network. It looks like you are testing with a Spark speedtest server. Try others, starting with ones that are near that one (in network nearness = ping times), but randomly try some others. Do not limit yourself to Spark servers - 2Degrees has some good ones, but for me, when I occasionally bother to try them, the best one is usually one on a small ISP I used to use, World-Net in Auckland. Speedtest servers are often under provisioned and overloaded, both in their CPUs and network bandwidth, and their status can change fairly frequently. On top of that, the TCP/IP stack on Windows is quite bad compared to Linux, so you often get the best result by DVD/USB/PXE booting a live Linux image (eg Ubuntu or Fedora) and testing from there.

AndyT

163 posts

Master Geek

#2823359 2-Dec-2021 18:10

Thank you fe31nz.

I think I'll focus on the testing as you suggest and michaelmurphy suggested earlier in the thread.

On the rural 4G BB, I think the best approach is "if it ain't broke don't fix it"!

Thanks & regards,

AndyT

163 posts

Master Geek

#2825367 6-Dec-2021 21:50

A quick update ....

I ensured WNAT and IPSEC offloading are enabled
I disabled Traffic Analysis / DPI
I rebooted the ONT and ER-X
I tried a direct connection between the iMac and ONT but couldn't run the Speedtest app, which I didn't have time to resolve
I put a Spark configured HG659 into circuit, but couldn't get an internet connection, which I also didn't have time to resolve
I then connected an ageing Dell laptop running Win10 to the ethernet that normally feeds the iMac and, hey presto, consistent 860Mbps or more on several speed tests. Photo attached
I then reconnected the iMac and it returned 856Mbps, but only once before reverting to mid 600Mbps on repeat tests. Photo attached

So, I think this tells me that:

Spark fibre service is OK
the Cat6 reticulation from ONT to ER-X is OK
the ER-X config and Cat 6 reticulation to the iMac is OK
there's an issue with the ageing (mid 2011) iMac's ethernet

Presumably the system is receiving and running with 860Mbps notwithstanding the iMac not reading it properly? This would appear to be what happened the other day when I was seeing a really good wifi 650Mbps on the U6-LR when the ethernet speed was also being reported at 650Mbps, as if there was no wifi overhead.

I've had several ethernet ports stop working altogether on Apple and Ubiquiti gear before, but can gigabit ports "go slow" but still perform in other respects?

The iMac's days are numbered but it's done excellent service and survived the CHC EQs, and the AMD graphics card oven-bake workaround to give it another couple of years active life, so I'm not complaining!

Thanks & regards,

AndyT

163 posts

Master Geek

#2826053 7-Dec-2021 16:24

Turns out that there are many, many reports on the web (e.g. Macrumors) about slow "gigabit" ethernet on Macs and Mac OSX, particularly older Macs like mine.

RunningMan

8955 posts

Uber Geek

#2826061 7-Dec-2021 16:37

AndyT: [snip]

there's an issue with the ageing (mid 2011) iMac's ethernet

Using a ten year old computer that's most recent OS is from 2017 probably should have been mentioned a little earlier in the troubleshooting.

Quic Broadband

Move to New Zealand's best fibre broadband service (affiliate link). Free setup code: R587125ERQ6VE. Note that to use Quic Broadband you must be comfortable with configuring your own router.

AndyT

163 posts

Master Geek

#2826313 7-Dec-2021 22:46

You're right, and I apologise. It just hadn't occurred to me until I set about logically testing the system yesterday.

Thanks & regards,

AndyT

163 posts

Master Geek

#2829201 9-Dec-2021 08:55

Having done some more testing, it seems the ethernet speed is not dependent on the age of the hardware or operating system.

Last night I tested the old iMac, the old Dell laptop and a current HP Elite X2 1013 G3 business laptop against each other multiple time in quick succession on exactly the same ethernet cable feed from the Spark fibre / ONT / Cat 6 reticulation / ER-X / USW with the results:

1. iMac: Generally in range 650Mbps - 690Mbps

2. Dell laptop: Generally in range 850Mbps to 870Mbps

3. HP Elite X2: Generally in range 460Mbps to 670Mbps

I was expecting the HP Elite to max out at 940Mbps or so, but not the case. But it did convince me that the issue does not lie with the Spark fibre / ONT / Cat 6 reticulation / ER-X / USW.

I think this is case closed on this thread, and probably a new thread on ethernet speed variability issues in general.

Thanks & regards,

RunningMan

8955 posts

Uber Geek

#2829535 9-Dec-2021 15:21

AndyT:

Having done some more testing, it seems the ethernet speed is not dependent on the age of the hardware or operating system.

Last night I tested the old iMac, the old Dell laptop and a current HP Elite X2 1013 G3 business laptop against each other multiple time in quick succession on exactly the same ethernet cable feed from the Spark fibre / ONT / Cat 6 reticulation / ER-X / USW with the results:

1. iMac: Generally in range 650Mbps - 690Mbps

2. Dell laptop: Generally in range 850Mbps to 870Mbps

3. HP Elite X2: Generally in range 460Mbps to 670Mbps

Huh? That proves almost exactly the opposite. The hardware/software spec is having a significant impact, with one client seeing performance about half one of the others. It may not be the age per se but there will be a correlation between age and specs (therefore performance). There are also numerous issues noted with a number of Windows ethernet drivers that have a negative impact on throughput.

1 | 2