Optimised EdgeRouter X Configuration for Spark UFB PPPoE Internet Connection

Forums › LAN (ethernet/Wifi/routers/Bluetooth) › Optimised EdgeRouter X Configuration for Spark UFB PPPoE Internet Connection

AndyT

166 posts

Master Geek
+1 received by user: 10

#290684 27-Nov-2021 20:58

I posted a bit of story on GZ in early January this year about the relatively slow speeds I was experiencing on a UFB PPPoE connection with an ER-X in front of a Spark Smart Modem & Mesh unit as dumbed-down WAPs (a long story as to why the system was arranged this way), which kind of stalled with the knowledge that the ER-X doesn't quite have the chops to run a gigabit connection.

Anyway, I have now removed the Smart Modem and Mesh unit from the system and invested in a Unifi U6-LR WAP and am keen to eek the very max out of the ER-X.

So, does anyone have a config for the ER-X on UFB PPPoE that they could share, that would perhaps get close to 900+ Mbps as opposed to the 650 to 700Mbps that I currently achieve?

The ER-X is currently configured per Spark's recommendation for third party routers, so may not be fully "optimised." It's interesting that when I look at the metrics in EdgeMax OS, it says the CPU usage is generally very low at 1% or 2%, although RAM usage is higher at a usual 20% to 30%, so the ER-X wouldn't seem to be under stress and could perhaps be "pushed" a little harder?

Thanks & regards,

Rgds Andy T

1 | 2

1303 posts

Uber Geek
+1 received by user: 430

#2820380 27-Nov-2021 23:42

The bottleneck in getting full 1 gigabit routing is normally in two places, the actual routing, and the PPPoE connection. Of the two, the one usually causing the most slowdown is PPPoE.

To get a ER-X to do full speed on PPPoE, you have to have it offloading the PPPoE processing and packet routing on to its builtin hardware, rather than doing it on the CPU. This page explains it:

https://help.ui.com/hc/en-us/articles/115006567467-EdgeRouter-Hardware-Offloading

My understanding is that with hwnat enabled, an ER-X should do a PPPoE connection at gigabit speed, but I only have ERLs and an ER4, so I can not say for sure. With my ERLs, if I do not have hardware offloading enabled, and I have my usual (large) set of firewall rules, I would expect throughput to be well less than 200 Mbit/s. My ER4 has a much bigger CPU, but I expect it would still struggle to do 1 gigabit with its offloading switched off. PCs with their much more powerful CPUs running PPPoE in pFsense often struggle to do 1 gigabit PPPoE. Part of this is that PPPoE software normally only runs on one CPU core, so that a PC or router with multiple cores can not make use of them for PPPoE and one core gets maxed out doing that job.

The problem with PPPoE is that each packet has to be modified to add or remove the PPP headers, and that usually means that the packet has to be copied to a different buffer. Any router that is copying all the packets going through it will struggle - the normal process for routing a packet is to keep the packet in a buffer and just add a pointer to that buffer on to the queue for the outbound port.

There is an additional complication with offloading in routers - there are usually features in routers that can not be offloaded. So if you have enabled one of those features, even though you have set the option to offload the packets, they will not actually be offloaded. So when you set an option like hwnat in your configuration, after you have applied the setting, you need to check the status of offloading to see if it is actually being used. There should be an ER-X command something like "show hwnat" to show the actual offloading status.

The feature that people normally want to use that causes offloading to be disabled is Quality of Service (QoS), where the router gives priority to some types of packets over others based on rules you set up. QoS is generally not needed for a gigabit connection, but if you are moving up from a much slower connection (such as ADSL), you may have had QoS enabled to help high priority traffic get through on time. For example, QoS is usually needed on a slow ADSL connection if you want VOIP to work properly. Or you may have set your bittorrent packets to be lower priority. So if you have been using QoS, check your configuration and disable it now. There are some sorts of QoS which do work with offloading, generally just getting the router to read and obey the DSCP priority bits in the IP headers, rather than modifying them. But it varies from one router to another. In my ERLs and ER4, DSCP does work with offloading, but the ER-X is based on a MediaTek chipset rather than the Cavium chipsets used in ERLs and ER4s, so it may be different.

insane

3329 posts

Uber Geek
+1 received by user: 1009

ID Verified

Trusted

2degrees

Subscriber

#2820390 28-Nov-2021 01:36

If you're open to changing ISPs, Vodafone supports DHCP, so no need for PPPoE. Others may too.

Having said that, havn't noticed any real world difference using a unify USG. Unless perhaps running speedtests and bragging to your mates is your thing.

michaelmurfy

meow
13664 posts

Uber Geek
+1 received by user: 11069

Moderator

ID Verified

Trusted

Lifetime subscriber

#2820391 28-Nov-2021 01:56

The Edgerouter X supports Gigabit over PPPoE totally fine - as per your other thread it does indeed appear you've got a rather messy setup. Speedtests are not accurate and it honestly sounds like you're expecting WiFi to give you Gigabit speeds which, it won't. You need to be testing over Gigabit Ethernet.

To be honest I don't see why you can't just use the Spark Smart Modem - what do you really need from the Edgerouter specifically that can't be done by a more basic router?

There is nothing to optimize. The fact you're only seeing the CPU bump up a little bit under load means you've got HWNAT successfully enabled. If you didn't have HWNAT successfully activated you'll note the CPU will ramp up to 100% and you'll be limited to about 500Mbit. As this is not the case, I have no doubt the router is performing fine so the problem is downstream on your network or it is a problem with how you're testing.

Michael Murphy | https://murfy.nz
Referral Links: Quic Broadband (use R122101E7CV7Q for free setup)

Are you happy with what you get from Geekzone? Please consider supporting us by subscribing.
Opinions are my own and not the views of my employer.

AndyT

166 posts

Master Geek
+1 received by user: 10

#2820485 28-Nov-2021 11:56

Thank you all.

fe31NZ: I'm offloaded to the max per UI advisory articles

insane: I must admit I hadn't considered switching ISP as we're too "invested" in Spark to make it an easy change, and generally speaking I'm OK with Spark

michaelmurphy: I agree the old setup was messy, but born out of almost necessity just before Christmas last year when there was a problem with Spark service (per my GZ posting back then). I knee-jerked (under pre-Christmas domestic pressure) and bought and installed an ER-X as I'm familiar with it from elsewhere (see below) and all's been good since. The Smart Modem and Mesh units were dumbed down as WAPs only and worked very well as such, but (i) couldn't do VLANs, and (ii) couldn't do Guest networks if the Mesh unit was installed. So I eventually and recently replaced them with the single U6-LR. The system is now as simple as it gets: ONT Cat6 to ER-X eth; ER-X eth1 Cat6 to USW; USW Cat6 + POE to UAP U6-LR. I'm aware wifi will never give full gigabit but it's now very good, reliable and evenly distributed. The 650 to 700 Mbps I referred to is based on Speedtest app on an iMac connected via Cat6 to the ER-X, which is what I'm keen to tweak, if there's benefit in doing so. On your last point, it's now a really simple system with all offloading in place, reticulated with new Cat6, so the issue would appear to be in testing, unless firewalls are slowing things down.

As a backdrop to all this, we have two properties. I'd happily been using Spark modem / router combos for fifteen years on ADSL in a SI rural location but Covid and WFH put paid to all that in April 2020 when ADSL became so unreliable from a business perspective under pressure of everyone WFH that changes had to be made. I switched to Spark wireless rural broadband which is far better but needed tweaking and supplementing to get the best out of it at the rural property - hence I started to get into networking and with GoWifi HQ just around the corner, I got into Ubiquiti products, since when it's all become an interesting pastime / hobby. The original posting relates to the system at our bach, at which I'm emulating the system at the rural property, the difference being the 100 year old bach has gigabit fibre broadband!

Rgds Andy T

AndyT

166 posts

Master Geek
+1 received by user: 10

#2820643 28-Nov-2021 15:23

A further thought / clarification.....

How do the Spark stock modems achieve their near gigabit speeds given that all I read is that for technical reasons (beyond my understanding) PPPoE makes gigabit pretty much unachievable except under controlled lab conditions?

Is it simply a question of CPU "brute force"? Or config overheads that you get with ERs but not so much in ISP stock modems?

If "yes" to brute force, and assuming I stick with an ISP with PPPoE not DHCP, would upgrading from the ER-X to the ER4 give a meaningful and noticeable boost in performance, or do I need to give up the ability to configure a "prosumer" router and switch back to uncomplicated stock equipment to get the higher speeds?

Thanks & regards,

Rgds Andy T

michaelmurfy

meow
13664 posts

Uber Geek
+1 received by user: 11069

Moderator

ID Verified

Trusted

Lifetime subscriber

#2820652 28-Nov-2021 16:54

ISP’s provide routers that work with all their plans and are stable - the smart modem still works well. Given everything you’ve said above I still have no clue why you need the Edgerouter as the Smart Modem also has guest mode capability…

Lastly as stated the Edgerouter X can do Gigabit PPPoE.

I seriously don’t understand what you’re doing here. I’m fully blaming your client or the way you’re testing.

Samsung

Shop now on Samsung phones, tablets, TVs and more (affiliate link).

AndyT

166 posts

Master Geek
+1 received by user: 10

#2820705 28-Nov-2021 17:35

I'm probably not explaining myself well, but thanks you for your advice anyway, and I'll review the client config and how I'm going about testing the wired ethernet speeds.

Thanks & regards,

Rgds Andy T

jbrook3708

34 posts

Geek
+1 received by user: 12

ID Verified

#2820764 28-Nov-2021 21:34

My ER-X is performing fine on Spark Fibre Max tonight:

Huawei ONT (white) -> ER-X -> US-8 -> PC

I assume you have cast your eye over michaelmurfy's excellent ER setup guide? My configuration is based on this with some VLANs and routing rules added.

Ubiquiti Edgerouter Tutorial

AndyT

166 posts

Master Geek
+1 received by user: 10

#2820781 28-Nov-2021 23:27

Thank you brook3708, that's encouraging.

Yep, I have been through michaelmurphy's ER setup quite a while back (+Toasty Answers, Crosstalk Solutions, Willie Howe and lots of other YT tutorials), but I think I'm going to have to work through it thoroughly again as clearly Im doing something that's not quite right in the client config or testing. But as I say, it's really encouraging that you're getting such good results on Spark in CHC with virtually the same hardware, plus the normal VLANs and Firewalls.

Let's see how I get on. I'd not want to flash out $350 or so on an ER-4 to find out I'm no better off than with the ER-X, for want of a little research and reconfiguration on the the ER-X

Thanks & regards,

Rgds Andy T

michaelmurfy

meow
13664 posts

Uber Geek
+1 received by user: 11069

Moderator

ID Verified

Trusted

Lifetime subscriber

#2820784 29-Nov-2021 00:48

The best thing you can honestly do right now is put your Smart Modem back in and test via that. I’ve had many Edgerouter X’s on Gigabit connections and all worked well.

I also would like to add that I’ll no-longer recommend the Edgerouter line due to the fact it’s mostly abandoned now (look at software updates). I honestly don’t know what you’re doing with it as you have not really explained this but either Mikrotik or the Dream Machine may be the best fit for you.

AndyT

166 posts

Master Geek
+1 received by user: 10

#2821229 29-Nov-2021 20:14

OK jbrook3708 ..... I checked out the ER-X config vs. michaelmurphy's ER tutorial and it's all in alignment ..... except on the following two points which are technically beyond me! :

MTU Values

MTU is set at a default value of 1500 on each LAN interface, but at 1492 on the WAN interface as default for PPPoE. Then as an option, TCP MSS Clamping is set to a value of 1452 - not sure where that value came from as the default is 1412.
Should these all be the same value? And is the option for TCP MSS Clamping necessary at all for a Spark UFB PPPoE connection?

IPv6

Under the IPv6 heading & PPPoE Configuration, the tutorial has a number of deletions, set up commands and a default route. I don't use IPv6 at all, so unless it's enabled in the default ER-X config and therefore necessary, can I safely ignore this section of the tutorial?

Thanks & regards,

Rgds Andy T

Dell

Shop now for Dell laptops and other devices (affiliate link).

michaelmurfy

meow
13664 posts

Uber Geek
+1 received by user: 11069

Moderator

ID Verified

Trusted

Lifetime subscriber

#2821234 29-Nov-2021 20:24

MTU Value should be 1500 for the PPPoE interface, 1508 for the VLAN and 1512 for the Ethernet interface. Disable TCP MSS Clamping.

Spark also still do not have IPv6 so don't worry about that configuration.

Also, as a troubleshooting step as I've mentioned a few times above - configure your Smart Modem and test that as the router for additional troubleshooting as this would have answered your own question very early on.

AndyT

166 posts

Master Geek
+1 received by user: 10

#2821259 29-Nov-2021 21:28

Many thanks michaelmurphy.

I'll plug those MTU values in and see what happens. And I'll run a standalone test on the Smartmodem next weekend, but I'm pretty sure it will return 940Mbps or thereabouts, as it used to, on bare bones wired ethernet (not wifi). I agree with you, its an excellent unit, as is the associated Smartmesh unit, with really good wifi, but for various reasons I'd moved to Ubiquiti gear and am now trying to get the best out of it. It sounds like the MTU adjustments might do the trick.

Thanks & regards,

Rgds Andy T

fe31nz

1303 posts

Uber Geek
+1 received by user: 430

#2821294 30-Nov-2021 01:57

Those MTU values are not quite right. The PPPoE is 1500 and VLAN 10 is 1508, but Ethernet is also 1508, not 1512. I checked my ER4 config to make sure I was remembering it correctly.

AndyT

166 posts

Master Geek
+1 received by user: 10

#2821324 30-Nov-2021 07:44

Thank you fe31nz.

Just to clarify, does the MTU value of 1508 get applied to (i) only the parent ethernet interface or (ii) all switched VLAN interfaces on the ER-X?

Thanks & regards,

Rgds Andy T

1 | 2