Optimizing Gigabit Fibre (UFB) for pfSense

Forums › New Zealand Broadband › Optimizing Gigabit Fibre (UFB) for pfSense

freakalad

236 posts

Master Geek
+1 received by user: 10

#243623 21-Dec-2018 12:39

Kia Ora,
Been a while....

I'm a long-time pfSense users, running my 64-bit stack on a PC Engines APU (2 I think?) embedded board.

I've recently switched from 100 Mpbs to 1 Gbps fibre connection.
Prior to the upgrade, I was getting a solid 100 meg down (sometimes edging a margin-of-error over), testing via linetest.nz as per ISP's recommendation.
But since the witch-up, I'm only getting around 30-40% of attainable, which is a far cry from what I expect.

Using the ISP's NetComm router, I'm seeing linetest.net speeds in the mid-90%, but on my other gear I've configured similarly (PPPoE with VLAN 10), my speeds consistently stay under 50%. I can only infer that there's some QoS/optimization "magic sauce" on their box I've yet to discover.

Some of my other testing gear include:

pfSense - my preferred method, for obvious reasons
A DrayTek Vigor 2860n
Fedora laptop's GBE naked on ONT

I've looked into & tried some of the system tunables optimisations put forward in the pfSense documentation, but nothing I've tried (in several combinations) make a significant difference.

Anyone else with a similar setup, could you please offer up some suggestions that might edge me towards the improvements I'm looking for?

Thanks

1 | 2 | 3

4644 posts

Uber Geek
+1 received by user: 2621

Trusted

#2148799 21-Dec-2018 12:45

Google for [device name here] nat performance. Google suggests it only does single core routing.

You're not on Atlantis anymore, Duncan Idaho.

hio77

'That VDSL Cat'
13044 posts

Uber Geek
+1 received by user: 3901

ID Verified

Trusted

Lizard Networks

Subscriber

#2148804 21-Dec-2018 12:53

I'd look at tuning the kernal.

https://www.netgate.com/docs/pfsense/hardware/tuning-and-troubleshooting-network-cards.html#pppoe-with-multi-queue-nics

Being that your RSP supplied modem is a netcom, I'd suspect your on slingshot.

They do PPPoE so that's your biggest issue here.

Last i checked, the implementation in BSD that pfsense use, is single threaded. so you really need quite a high clocked cpu rather than large counts of cores.

Newer cpus with the IPC gains will obviously assist there too.

If your adding shaping or queues, you would probably take a bigger hit too.

Could possibly see if DHCP is supported too, It is on orcon and as i understand it the core among vocus is shared now.. @Sounddude should be able to confirm.

#include <std_disclaimer>

Any comments made are personal opinion and do not reflect directly on the position my current or past employers may have.

MadEngineer

4644 posts

Uber Geek
+1 received by user: 2621

Trusted

#2148809 21-Dec-2018 12:59

Definitely DHCP, VLAN 10 on Orcon.

You're not on Atlantis anymore, Duncan Idaho.

muppet

2681 posts

Uber Geek
+1 received by user: 1734

Trusted

#2148825 21-Dec-2018 13:25

edit: hio77 already posted most of this, I'm a muppet.

The problem is PPPoE on FreeBSD is single threaded.

Read this - then read this.

There's a few suggestions in there, but you probably aren't going to go too fast sorry.

To get Full Gigabit speed you need a very capable device.

And even then, you going to run a RJ45 around everywhere, or use Wireless? Because if it's Wireless, you're not going to get much more than 300-400Mb/s anyway.

(These reasons are why I personally consider 1Gb/s for the home the same as a Ferrari. Sounds nice and flash but in the end you're only allowed to go the same speed as everyone else anyway)

Eitsop

584 posts

Ultimate Geek
+1 received by user: 153

ID Verified

#2148857 21-Dec-2018 14:25

@freakalad are you saying certain NICs have a problem on pfSense or all pfSenses will have this issue?

freakalad

236 posts

Master Geek
+1 received by user: 10

#2148869 21-Dec-2018 14:33

Thanks for the pointers, folks.

It seems the AMD G-T40E PCU might not be up to the task - 2 cores

Set values @ /boot/loader.conf.local for:

net.isr.dispatch=deferred
net.isr.maxthreads=4
net.isr.numthreads=4

Gains was too modest - <10% maybe

Fall-back plan may be to use the modem in bridging mode - use the NetComm to do the PPPoE 'dialup' & my pfSense to manage my network

FLOSS'er, aspiring Maker

Lenovo

Shop now for Lenovo laptops and other devices (affiliate link).

freakalad

236 posts

Master Geek
+1 received by user: 10

#2149046 21-Dec-2018 19:48

attewell:

@freakalad are you saying certain NICs have a problem on pfSense or all pfSenses will have this issue?

I think @muppet has it right above - in short, it'll probably be virtually all pfSense devices on somewhat lower-end hardware; certainly seems to be the case on the PC Engines boxes I've always been utilising.

From what I can tell, the hardware (CPU in particular) needs to be multithreading/hyperthreading, due to the PPPoE implementation.

IF you have the hardware capable of taking the strain, then there may be a few tweaks to be had (see above) to force pfSense into making different use of resources, but my own knowledge on that nuanced discussion is too limited to give a confident answer.

The TL;RD seems to be we may need for either driver rewrites and/or different implementations in the upstream kernel before a non-hacky solution is to be found.

"Unfortunately, RSS is usually capable of hashing IPv4 and IPv4 traffic (L3+L4). All other traffic like PPPoE or MPLS or .. is usually received by queue 0."
https://wiki.freebsd.org/NetworkPerformanceTuning

IF anyone stumbles across this thread in a year or few down the line, looking for a solution to this (seemingly not-uncommon) problem, and it HAS been fixed upstream (or simply not an issue anymore due to a release upgrade), please be so kind as to notify us.

ripdog

548 posts

Ultimate Geek
+1 received by user: 373
Inactive user

#2149122 22-Dec-2018 05:05

Really stupid question, but have you checked the router's CPU usage graph while running a speedtest? That would prove or rule out a CPU bottleneck.

hio77

'That VDSL Cat'
13044 posts

Uber Geek
+1 received by user: 3901

ID Verified

Trusted

Lizard Networks

Subscriber

#2149130 22-Dec-2018 07:33

Yeah that cpu is really on the weak side....

#include <std_disclaimer>

Any comments made are personal opinion and do not reflect directly on the position my current or past employers may have.

Eitsop

584 posts

Ultimate Geek
+1 received by user: 153

ID Verified

#2149409 22-Dec-2018 19:08

hio77: Yeah that cpu is really on the weak side....

Is my CPU ok? Will I get full gigabit speed

Intel(R) Core(TM) i5-5250U CPU @ 1.60GHz 4gb ram 60gb ssd
4 CPUs: 1 package(s) x 2 core(s) x 2 hardware threads

sbiddle

30853 posts

Uber Geek
+1 received by user: 9998

Retired Mod

Trusted

Biddle Corp

Lifetime subscriber

#2149411 22-Dec-2018 19:17

attewell:
hio77: Yeah that cpu is really on the weak side....

Is my CPU ok? Will I get full gigabit speed

Intel(R) Core(TM) i5-5250U CPU @ 1.60GHz 4gb ram 60gb ssd
4 CPUs: 1 package(s) x 2 core(s) x 2 hardware threads

Unlike if you're using PPPoE

Haver you actually read the posts above that explain what the issues are around PPPoE?

pfSense is a good product - but IMHO it's major issues with architecture mean it's not something that should be used if you have a fast connection and/or simply want your speedtests to show Gigabit line speed, because it will struggle.

Apple TV

Stream your favourite shows now on Apple TV (affiliate link).

Eitsop

584 posts

Ultimate Geek
+1 received by user: 153

ID Verified

#2149413 22-Dec-2018 19:23

I am with snap/2d who only use PPPoE right?

So what kind of PC do I need. What is better then pfSense?

sparkz25

750 posts

Ultimate Geek
+1 received by user: 284
Inactive user

#2149419 22-Dec-2018 19:53

after some testing and alot of homework afew of us took the plunge on these little pc's

https://www.aliexpress.com/item/Free-Shipping-4-Gigabit-LAN-ports-Mini-PC-Celeron-3215U-Core-i3-Core-i5-WIFI-using/32829499825.html?spm=a2g0s.9042311.0.0.63fc4c4dK9thux

We chose the I5-5200u model because it also has AES which helps with the VPN

I have been using this for about 6 months on gig fiber with 4 vpns on the device, and 3 of the vpns have the exact same box at the end, we have managed to hit speeds of 40MB's over our vpns which is awesome compared to our old links of 10MB, we opted for opnsense as the os instead of pfsense

I have never seen it use 100% Cpu ever and it chugs along smoothly on 4gb of ram nicely, you can see in the pic where a speed test has been performed and how much cpu it used at the time, peaked at 60% on speed test.net and the rest were on fast.com

Eitsop

584 posts

Ultimate Geek
+1 received by user: 153

ID Verified

#2149433 22-Dec-2018 20:20

@sparkz25 what speed do you get outside VPN

sparkz25

750 posts

Ultimate Geek
+1 received by user: 284
Inactive user

#2149434 22-Dec-2018 20:22

attewell: @sparkz25 what speed do you get outside VPN

750-850 pending on the time of day and so on

1 | 2 | 3