Ubiquiti Edgerouter Tutorial

Forums › LAN (ethernet/Wifi/routers/Bluetooth) › Ubiquiti Edgerouter Tutorial

1 | ... | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37

Mr Snotty
8620 posts

Uber Geek
+1 received by user: 4514

Moderator

Trusted

Lifetime subscriber

# 2227053 28-Apr-2019 22:15

aquatarkus:

Is this the right thread for a question about setting priorities for network traffic when there is an Edgerouter Lite in the network?

Don't do this! Especially so since you're on a Gigabit plan. It sounds like offloading is not correctly enabled on your Edgerouter Lite as on my network when my server is uploading backups to Crashplan I don't notice it.

QoS disables offloading and will slow things down, there are other things that will disable Offloading also. Ensure you're using Ethernet where you can, have offloading enabled and have also ensured you're not using any features that disable offloading: https://help.ubnt.com/hc/en-us/articles/115006567467-EdgeRouter-Hardware-Offloading

Michael Murphy | https://murfy.nz
A quick guide to picking the right ISP | The Router Guide | Community UniFi Cloud Controller | Ubiquiti Edgerouter Tutorial

fe31nz

437 posts

Ultimate Geek
+1 received by user: 92

# 2227056 28-Apr-2019 22:37

You can set traffic priorities in an ERL, but doing that will likely cause the CPU to have to process every packet in order to rewrite the priority bits. Which means there will not be any hardware offloading, so you will not get anywhere near gigbit speed any more. And enabling QoS processing will also turn off hardware offloading.

The ONT/fibre connection does have the ability to handle some prioritisation. It has a "committed bit rate" (CBR) defined that says that no matter what congestion happens on their network, you will always get at least the CBR level of traffic available to your connection. It is a long time since I looked at the specifications for the CBR, but I think it was 1.25 Mbit/s in each direction on my gigabit connection. Not huge, but easily enough to carry all your VOIP traffic and some other important things like DNS lookups. All your other bandwidth is subject to contention with other users of the network. If you want your packets on the fibre network to use your CBR bandwidth, they have to be tagged with the correct DSCP bits when handed over to the fibre network at either end. So I have this command in my ERL config:

set interfaces ethernet eth0 vif 10 egress-qos '1:5 2:5 3:5 4:5 6:5 7:5'

What that does is translate the DSCP bits that I have on my VOIP packets to the correct value for the ONT to send them using my CBR bandwidth. I have never been able to properly test this to see if it actually works though, as the actual rate of contention on the fibre network is never high enough that it would make a difference to be using CBR bandwidth. Doing this does not affect hardware offloading as far as I know. But any other sort of priority processing that an ERL can do that I can think of will disable offloading, so must be avoided at all costs on a gigabit connection.

Switches are a different ball game. If you have an intelligent switch, it will very likely be able to do prioritisation of your traffic based on the DSCP bits. Some switches may also be able to set the DSCP bits for you, based on appropriate rules that may be able to distinguish your backup packets. I am unsure if switches can do this in hardware, or if they need to use their CPU to process the packets. If it can be done in hardware, then there will be no cost, but if the CPU needs to be used, then you are again effectively disabling hardware offloading and there will be a (probably significant) speed penalty. If you have a switch with the correct capabilities, it would be worth trying.

What backup software are you using? Does it offer an option to set the DSCP bits? Setting them at source is always the best idea. Then you can get an intelligent switch if necessary and set it to do the appropriate priority processing. My Edgerouter ES24 Lite has this capability of doing QoS proessing in hardware, although I am not actually using it. It is a pretty standard feature on "business" grade switches.

Jedi

9 posts

Wannabe Geek

# 2227060 28-Apr-2019 23:18

I have a EdgeMax and EdgeSwitch 8. From what I have been reading on here my internet speed is light years away from what is possible.

Any help with this would be appreciated.

aquatarkus

53 posts

Master Geek
+1 received by user: 5

# 2227066 28-Apr-2019 23:54

Thanks for the suggestions.

After reading them I realised that I had been less than clear about what was contending with what.

I have a gigabit fibre connection and speeds there are good. Not Jedi's 104/14 but more like 800/450. The house has both wifi and gigabit ethernet throughout; everything is using a wired connection except phones and tablets, and a chromecast device on one of the TVs. My initial theory was that the internal network was struggling with the bandwidth when backup and streaming processes were running together.

Just as a test I started ad-hoc backups from two laptops and looked at the TV performance streaming Netflix vs using Plex to play back mpegs from the NAS. The performance degradation was minimal with Netflix and significant with Plex. So actually I think the problem is not a network bottleneck but the performance of the NAS. I'm relieved to find that is the case.

BTW the backup software is EaseUS, and the NAS's both have dual network connections.

This means the question has got nothing to do with edgerouters and doesn't belong in this thread. Sorry for the interruption.

cheers
T

fe31nz

437 posts

Ultimate Geek
+1 received by user: 92

# 2227069 29-Apr-2019 01:04

Jedi:

I have a EdgeMax and EdgeSwitch 8. From what I have been reading on here my internet speed is light years away from what is possible.

Any help with this would be appreciated.

Your offloading is not set up properly. Connections to Spark need PPPoE over VLAN 10. So all your traffic to Spark is going over a VLAN, but you have not enabled hardware offloading of VLAN processing. I am not sure if you can turn it on via the GUI as I always use the CLI. The CLI commands you need are:

configure

set system offload ipv4 vlan

commit

save

exit

If you are using IPv6, you will also need to enable offloading for that too.

aquatarkus

53 posts

Master Geek
+1 received by user: 5

# 2227070 29-Apr-2019 01:41

Just out of interest I checked my offloading settings and they were the same as Jedi's. So I ran a speedtest, made the change suggested by fe31nz (which incidentally needs an 'enable' on the end of the 'set' command, restarted, and ran speedtest again.

Before the change: 800 down / 150 up
After the change: 410 down / 13 up

Oops.

So I went back to disabled vlan offloading and restarted.

Now my speedtest is 690 down / 14 up.

I can't say I'm happy with the results. I'd like my original upstream speeds back thank you very much!

michaelmurfy

Mr Snotty
8620 posts

Uber Geek
+1 received by user: 4514

Moderator

Trusted

Lifetime subscriber

# 2227079 29-Apr-2019 07:37

aquatarkus:

Just out of interest I checked my offloading settings and they were the same as Jedi's. So I ran a speedtest, made the change suggested by fe31nz (which incidentally needs an 'enable' on the end of the 'set' command, restarted, and ran speedtest again.

Before the change: 800 down / 150 up
After the change: 410 down / 13 up

Oops.

So I went back to disabled vlan offloading and restarted.

You've got something pretty funky in your configuration to do this. The default settings should be fine, but you also need to have VLAN offloading enabled.

Michael Murphy | https://murfy.nz
A quick guide to picking the right ISP | The Router Guide | Community UniFi Cloud Controller | Ubiquiti Edgerouter Tutorial

Lightbulb

50 posts

Geek

Lifetime subscriber

# 2227294 29-Apr-2019 12:02

Not sure whether this belongs in a separate thread - if so, apologies etc. I can repost if necessary.

I have had an edgerouter 4 for a couple of months. A couple of days ago I was 'messing about' with the NAT settings and I'm not sure whether I 'accidentally' deleted a standard destination rule from the standard edgerouter default setup.

Coincidently (or not) the internet has been disconnecting on a regular basis (both ethernet and wireless), but only since then. I have three 5G access points, 2 HG 659b's (both with DHCP disabled) and a Linksys router configured as an access point (DHCP disabled) - all using same SSID, encryption and password. Everything was all good until a couple of days ago when I (possibly) made some changes to the edgerouter setup.

If I reboot the HG659b's, it's all good for a short time - then the internet drops out again. However, leaving both HG659bs off and running with just the Linksys works fine - implying something wrong with theHG659s... maybe.

I've factory reset the two HG659s, to no avail.

A screen shot the NAT settings page is as follows:

Any help / comments appreciated.

aquatarkus

53 posts

Master Geek
+1 received by user: 5

# 2227600 29-Apr-2019 20:14

One person supports this post

Nothing funky here that I can see. Today I reverted to the original settings - all the defaults, with offload ipv4 vlan DISabled. Then I ran speedtest 4x each on several different machines. I have two subnets here; they are managed by a Logic Systems NUC running Ubuntu which is connected (once) to the ER-lite.

1 Ubuntu: average 226 down, 181 up
2 Fedora (subnet 1): 645 down, 303 up
3 Windows 10 (subnet 1): 264 down, 273 up
4 Windows 10 (subnet 2): 159 down, 127 up
5 Windows 10 (subnet 2): 260 down, 187 up

Then I ENabled vlan offloading again and rebooted everything

1 Ubuntu: average 221 down, 199 up
2 Fedora (subnet 1): 658 down, 308 up
3 Windows 10 (subnet 1): 194 down, 116 up
4 Windows 10 (subnet 2): 450 down, 51 up
5 Windows 10 (subnet 2): 355 down, 58 up

So there's minimal effect on the Linux machines, but the Windows machines are a bit of a dog's breakfast. Downloading 2 of the 3 are significantly better, but uploading all 3 of them are more than 50% worse. Is this expected? Is offloading optimal for download and sub-optimal for upload? Or is there something else going on?

I don't know why the Fedora machine is so much faster than the others - there's nothing special about its network adapter, and it's not the cabling because I see the same performance difference when I swap the connections of the Fedora and Windows machines. Would that all the machines could transfer that fast :-)

If all I did was download this would be fine, but that isn't the case.

nicmair

99 posts

Master Geek
+1 received by user: 5

# 2227621 29-Apr-2019 20:39

Lightbulb, my NAT settings are the same as what you have, so I don't think you have deleted anything.

fe31nz

437 posts

Ultimate Geek
+1 received by user: 92

# 2227658 29-Apr-2019 22:25

aquatarkus:

Nothing funky here that I can see. Today I reverted to the original settings - all the defaults, with offload ipv4 vlan DISabled. Then I ran speedtest 4x each on several different machines. I have two subnets here; they are managed by a Logic Systems NUC running Ubuntu which is connected (once) to the ER-lite.

1 Ubuntu: average 226 down, 181 up
2 Fedora (subnet 1): 645 down, 303 up
3 Windows 10 (subnet 1): 264 down, 273 up
4 Windows 10 (subnet 2): 159 down, 127 up
5 Windows 10 (subnet 2): 260 down, 187 up

Then I ENabled vlan offloading again and rebooted everything

1 Ubuntu: average 221 down, 199 up
2 Fedora (subnet 1): 658 down, 308 up
3 Windows 10 (subnet 1): 194 down, 116 up
4 Windows 10 (subnet 2): 450 down, 51 up
5 Windows 10 (subnet 2): 355 down, 58 up

So there's minimal effect on the Linux machines, but the Windows machines are a bit of a dog's breakfast. Downloading 2 of the 3 are significantly better, but uploading all 3 of them are more than 50% worse. Is this expected? Is offloading optimal for download and sub-optimal for upload? Or is there something else going on?

I don't know why the Fedora machine is so much faster than the others - there's nothing special about its network adapter, and it's not the cabling because I see the same performance difference when I swap the connections of the Fedora and Windows machines. Would that all the machines could transfer that fast :-)

If all I did was download this would be fine, but that isn't the case.

Hardware offloading is always best for all traffic. It avoids having the packet processed in the CPU. Hardware offloading is *always* faster. If the router CPU is extremely fast, then it can be possible that CPU routed traffic will be as fast as offloaded traffic, but that is rare except when the physical speed of the connection is low enough. If you are routing at above 100 Mbit/s, then you need hardware offloading because any affordable router will not have enough CPU to do it fast enough.

What you seem to be seeing is just how bad speed testing software is on Windows machines in particular. Linux seems to be much better at running speed tests, and we certainly know much more about how its networking works due to it all being open source. Messing about in your router is the wrong place to be looking when dealing with such inconsitent results coming from using Windows. Windows is the problem. Have you checked that the Windows network stack is set up properly? One thing that is often the cause of problems like this is the "QoS packet scheduler" or what ever name it is called now. That can be useful when you are running a 100 Mbit/s Ethernet network, but above that speed, it only causes problems and should be disabled. There are often also options on the Ethernet drivers to do hardware offloading there - you need to try with them on and with them off, as some drivers are so bad that using the hardware offloading actually slows down the traffic. There are a great number of tweaks that can be looked at. You can spend years trying to get the best numbers out of speed testing, and then find that in real life software use, the speeds are different again.

Beyond just some generic suggestions, we would need to know a lot about your network and the PCs to be able to diagnose the problems. So a different thread might be a good idea. The speed of the Ubuntu machine might be quite consistent, but it is consistently bad. There will be a reason for that, perhaps simply that it may be an ancient machine that is just that slow. But we would need details to be able to work it out.

One thing you really ought to try is to boot from a live CD or live USB version of Linux on your Windows machines and do speed tests from there. That avoids all the Windows nastiness, at least.

michaelmurfy

Mr Snotty
8620 posts

Uber Geek
+1 received by user: 4514

Moderator

Trusted

Lifetime subscriber

# 2227665 29-Apr-2019 23:10

@aquatarkus You mentioned that this is going via a NUC? Why is the NUC here? Is it bridging the 2 sides of your network?

If so, this is very likely your weak point and I don't understand why you've got it here in the first place as the Edgerouter can handle it, in hardware. If you've got 2 subnets you've got to remember the router (or NUC) will be handling ALL the traffic between those subnets. If those subnets are very chatty between each over then you're limited by the 1Gbit interface of the NUC / Edgerouter combined with your internet traffic on top of it. It sounds like this is where you're getting congestion.

I have no idea why you're doing it this way - there are better ways to do it (via interVLAN Routing on a smart switch as an example) that won't run into this limitation.

Remove the 2 segments of the network or test direct via the Edgerouter without the NUC or anything complex in the way and without any other devices connected.

Michael Murphy | https://murfy.nz
A quick guide to picking the right ISP | The Router Guide | Community UniFi Cloud Controller | Ubiquiti Edgerouter Tutorial

aquatarkus

53 posts

Master Geek
+1 received by user: 5

# 2227689 30-Apr-2019 01:21

The NUC is there because it has been there for years. It is running DNS and DHCP, and provides a gateway to a couple of overseas VPNs which I connect to for work. The two subnets are work and home; work needs to see home but home must not see work, or access the VPNs. The original code implementing all this was supplied to me by a dev team colleague in one of the overseas offices and it mirrors what he had set up for his analogous circumstances.

The Edgerouter is there because the Spark-supplied HG659b could not cope with the demands of our network; as soon as there were 25 devices connected (in total) it would lock up. This may have been a problem that Spark should have been able to rectify, but they weren't interested in doing so, so I bought a much superior piece of kit to replace it. And there's no doubt the ER-L is superior, maybe it's overkill. While the HG659b could do some of the things that the NUC does it could not do them all, so I kept all those functions on the one device instead of splitting them up.

You're telling me that the Edgerouter can replace the NUC and take over all its functions, which would be great, and in time I could make that change. However I don't believe that the NUC is responsible, at least not solely responsible, for the poor speeds I am seeing on the laptops and desktop. That's because when we first went to our current Fibre plan using the HG659b I was seeing speeds in the 800 down/400 up range on all the devices.

As per @fe31nz's suggestion I'm making a live USB for Fedora and I'll see what kind of speeds that reports tomorrow.

cheers
T

aquatarkus

53 posts

Master Geek
+1 received by user: 5

# 2227914 30-Apr-2019 12:21

I think this definitively confirms @fe31nz's point about Windows being the problem. When I booted the windows PCs off the fedora live usb there was a dramatic improvement (again these are averaged across several tests):

Windows: 323 down / 41 up
Fedora: 722 down / 312 up

Windows: 302 down / 75 up
Fedora: 756 down / 307 up

freakngeek

332 posts

Ultimate Geek
+1 received by user: 86

# 2230662 4-May-2019 08:54

One person supports this post

New EdgeRouter firmware 2.0.3 has been released [ New ] Options
5 hours ago

New release v2.0.3 is available here:

ER-X, ER-X-SFP and EP-R6: Not ready because we are still searching for root cause of random filesystem corruption
(SHA256:xxx)
ERLite-3 and ERPoe-5: https://dl.ubnt.com/firmwares/edgemax/v2.0.x/ER-e100.v2.0.3.5189348.tar
(SHA256:05480823a63ebb77a2c4e93e81a470d8721a2d162d37e98c5ba72627b0946e29)

ER-8, ERPro-8 and EP-R8: https://dl.ubnt.com/firmwares/edgemax/v2.0.x/ER-e200.v2.0.3.5189348.tar
(SHA256:fe6f3a761d03607dc5b72a1a291f31ce72332f8c0f6a444de03278689d5d0642)

ER-4, ER-6P and ER-12: https://dl.ubnt.com/firmwares/edgemax/v2.0.x/ER-e300.v2.0.3.5189356.tar
(SHA256:4abfd0d62eb843ecac609952862b2a4f9f41787dc8910b2578cd2e25037d582a)

ER-8-XG: https://dl.ubnt.com/firmwares/edgemax/v2.0.x/ER-e1000.v2.0.3.5189348.tar
(SHA256:0064928f8f2dbbd042595b8494542c65533a546f03a745d7c5ddc0ce684d65af)

Note: The ER-X/ER-X-SFP/EP-R6 has more limited storage, and in some cases, an upgrade may fail due to not enough space. If this happens, remove the old backup image first (using "delete system image" command, see here for more details) before doing an upgrade.

More details can be found in the release notes below. Please give it a try if you are interested in the new features/changes to help us test them so that we can get the release out sooner! Thanks very much!

Note 2: Version v2.0.2 was not released because we identified major flaw during internal beta testing.

Note 3: We do not release v2.0.3 for ER-X/ER-X-SFP/EP-R6 models because we still did not fix regression from v2.0.0 that is causing random filesystem corruption of ER-X. When running stress tests on lab routers we were able to reproduce this issue few times - it seems that SQUASHFS filesystem image somehow becomes corrupted when ER-X is low on RAM, is doing a lot swapping and kernel's OOM manager is killing userspace tasks. Fix for this issue is not yet available because it's very hard to reproduce and catch filesystem corruption bug. WIP

[Release Notes v2.0.3]

Changelog

Changes since v2.0.1

New features:

Enhancements and bug fixes:

[WebGUI] - Fix regression in v2.0.0 when ping in WebGUI did not work. Discussed here
[WebGUI] - Fix bad eth9 interface in Basic Setup Wizard on ER-X and ER-4. Discussed here
[WebGUI] - Add checkbox to toggle IPv6 functionality from WebGUI
[WebGUI] - Fix bug that caused WebGUI/UNMS to become inactive if aggressive network scanners opens lots of unauthorized sessions
[QoS] - Fix bug in v2.0.1 that caused Qdisc to be configured on all ER-X interfaces. Discussed here
[Firewall] - Fix bad firewall rules when MSS clamping is configured. Discussed here
[Firewall] - Fix regression in v2.0.0 that caused firewall section to disappear if rule had TCP_UDP protocol assigned to HTTP port. Discussed here
[FlowAccounting] - Fix netflow daemon crash if WAN interface is down for more than 1 minute. Discussed here
[Interface] - Fix bug when rx/tx counters of switch-port interfaces were not updated.
[Interface] - Fix regression in v2.0.0 when statically configured IPv6 addresses on ER-8-XG were missing after reboot
[OpenVPN] - Fix regression in v2.0.0 when client-specific options were ignored.Discussed here
[Bonding] - Fix bug when bonding did not pass traffic if switch-port interfaces are used on ER-12. Discussed here and here
[QoS] - Now transit packets will automatically bypass offloading if netflow or qos or suspend are configured
[IPv6] - Fix IPv6 bug when address auto-configuration for VLAN interfaces did not work when enabled.
[IPv6] - Fix regression in v2.0.0 when radvd did not work correctly when multiple router-advert were configured. Discussed here
[MDNS] - Fix bug when mdns service did not start if VTI interface was configured. Discussed here
[Routing] - Fix bug when 0.0.0.0/0 static route sometimes was not restored when nexthop address became reachable
[CLI] - Fix Perl error message when running "clear dhcp lease ip ..." CLI command. Discussed here
[TechSupport] - Fix bug when tech-support file was truncated if flow-accounting was configured
[TechSupport] - Add output of journalctl to tech-support file
[TechSupport] - Add runtime state of flow-accounting to tech-support file
[UNMS] - add support for configuring QoS and suspended clients via UNMS

Known issues:

Updated software components:

base-files 9.9+deb9u9
fping 3.15-1
dns-root-data 2019031302~deb9u1
libpam-systemd:mips 232-25+deb9u11
libssh2-1:mips 1.7.0-1+deb9u1
libsystemd0:mips 232-25+deb9u11
systemd 232-25+deb9u11
tzdata 2019a-0+deb9u1
udev 232-25+deb9u11

Bootloader enhancements and fixes:

This firmware will be published on main forum and UNMS in 1-2 weeks if it proves to be stable.

1 | ... | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37