Upload not getting much above 100Mbit international

Forums › New Zealand Broadband › Upload not getting much above 100Mbit international

gareth41

733 posts

Ultimate Geek

#298834 20-Jul-2022 12:40

Currently with Voyager, but this was also the case with Spark at the old house - both 1Gbit/500Mbit UFB

I can never seem to achieve anything greater than 100Mbit/s up unless its local traffic within NZ, have run several tests.

- Uploading a single 1GB file to my sitehost VPS, I can max out at almost 500mbit

- Uploading over DC++ to friends located in NZ and on UFB - gets upwards of 400mbit. Uploading overseas, even Australia maxes out at 100Mbit using multiple connections - can't achieve anything more than this

- Same again with bittorrent, local peers can max out at close 500mbit with a single connection, multiple overseas peers are needed to get close to 100mbit combined speed and nothing above that

- Speed tests are the same, running a speed test to various locations around the globe, I can never achieve anything more than 100Mbit up. Local its all fine and maxes out my line.

On the other hand, downloading is no problem. I can retrieve a file from a VPS in Germany at around 400 to 500Mbits/s, uploading to that same VPS requires multiple connections to get close to 100Mbit. I'm assuming there is bandwidth management in place at some point when packets are exiting NZ ? I also went as far as running speedtests from my Sitehost VPS and got similar results there too - so this doesn't appear to be ISP related. I can only conclude there is upstream bandwidth management in place where packets are existing NZ - possibly global-gateway?

BarTender

3530 posts

Uber Geek

ID Verified

Trusted

Lifetime subscriber

#2944207 20-Jul-2022 13:30

You probably need to setup a traffic policer to throttle the traffic to 480mbit as what will be happening is you will be exceeding the tcp window rate and dropping a packet then TCP will need to retry.

MikroTik or Unifi routers can do it. But not standard consumer grade ones.

Otherwise running a UDP VPN to a domestic host should improve things as the packet loss will be handled a lot more reliably

and

Sharesies

Trade NZ and US shares and funds with Sharesies (affiliate link).

gareth41

733 posts

Ultimate Geek

#2944279 20-Jul-2022 14:59

Thanks for the info. I use a Mikrotik 3011UiAS, so will give this a try.

fe31nz

1085 posts

Uber Geek

#2944588 21-Jul-2022 00:32

If you are testing using a Windows machine, you may be running up against problems with the Windows TCP stack - it does not work well for larger ping times as it does not seem to adjust the window settings to cope with needing to send much more data before requiring an ACK to be received. Try a modern Linux kernel which should do this properly, but it normally takes a little while to ramp up as it increases the window size.

gareth41

733 posts

Ultimate Geek

#2945004 21-Jul-2022 23:32

fe31nz:

If you are testing using a Windows machine, you may be running up against problems with the Windows TCP stack - it does not work well for larger ping times as it does not seem to adjust the window settings to cope with needing to send much more data before requiring an ACK to be received. Try a modern Linux kernel which should do this properly, but it normally takes a little while to ramp up as it increases the window size.

I did more testing this evening, completely eliminated the Mikrotik router and connected a Linux box directly to the ONT running Ubuntu 20.04. It's actually an old Dell PowerEdge server I had sitting around. Tagged vlan 10 and setup pppoe. Ran multiple speedtests to London and Amsterdam

Here they are:

https://www.speedtest.net/result/c/cf2921b9-3055-4c8c-8907-a012136df45a

https://www.speedtest.net/result/c/6e39c256-8ebd-4445-a193-dd6ae56aecc9

Edit: Here's the London test

https://www.speedtest.net/result/c/040c0cf5-1bdf-4907-975d-2c209543e5e9

The upload maxes out just short of 50Mbit while the download is more than 400Mbit. The results were fairly consistent every time, eg not fluctuating from test to test.

I also tried tweaking various tcp/ip params to increase buffer limits etc... made no difference to the upload

/proc/sys/net/core/rmem_max = 16777216
/proc/sys/net/core/wmem_max = 8388608
/proc/sys/net/core/rmem_default = 16777216
/proc/sys/net/core/wmem_default = 8388608
/proc/sys/net/core/optmem_max = 40960
/proc/sys/net/ipv4/tcp_rmem = 4096 87380 16777216
/proc/sys/net/ipv4/tcp_wmem = 4096 65536 8388608

What I did notice was the data rate ramping up - upload starts off slow and ramps up quite fast then suddenly hits eg 48Mbit at a third of the way through the upload test and stays there. The download on the other hand, ramps up fast but keeps ramping up faster and faster right through the whole duration of the download test.

yitz

1874 posts

Uber Geek

#2945007 22-Jul-2022 00:25

At 182 ms I think you have the wrong London...

Downloads going up would be expected as you are running the multi server test, I'm not sure upload is ever multi server?

What congestion control algorithm are you running in /proc/sys/net/ipv4/tcp_congestion_control

Could be worth installing TCP BBR a try...

deadlyllama

1221 posts

Uber Geek

Trusted

#2945027 22-Jul-2022 08:01

TCP BBR will make a big difference.

gareth41

733 posts

Ultimate Geek

#2945242 22-Jul-2022 13:36

I installed / enabled TCP BBR and it made a huge difference!

https://www.speedtest.net/result/c/6ef14213-80a1-4d0e-8b80-252ea89d21f8

https://www.speedtest.net/result/c/70a2065d-997e-4e79-a6ee-ad8ea55b5624

I checked tcp_congestion_control before making the changes and it was set to cubic

I followed the steps on this github page:
https://gist.github.com/xterat/cefccb42f7d4b2055368ebb00454861f

Basically check the kernel is 4.9 or newer

Then run the following commands:

modprobe tcp_bbr echo "tcp_bbr" >> /etc/modules-load.d/modules.conf
echo "net.core.default_qdisc=fq" >> /etc/sysctl.conf
echo "net.ipv4.tcp_congestion_control=bbr" >> /etc/sysctl.conf
sysctl -p

Check that its been enabled:

sysctl net.ipv4.tcp_available_congestion_control
sysctl net.ipv4.tcp_congestion_control
lsmod | grep bbr

Thanks for the help everyone

fe31nz

1085 posts

Uber Geek

#2945509 22-Jul-2022 23:06

I am not familiar with old Dell servers, so I do not know how good their CPUs are. But there is a problem with doing PPPoE connections using just a CPU. All the PPPoE software I have met is single threaded and does not use multiple CPU cores if they are available. So the PPPoE performance of a CPU depends on the single core performance, and on older CPUs that often was not enough to do a full gigabit connection. Routers (even really good ones) generally have much smaller CPUs than PCs, and as a result of that almost invariably do their PPPoE using offloading hardware. I am pretty sure Mikrotik does that. So for best performance, it would be better to use the Mikrotik router to do the PPPoE, rather than an older CPU. And make sure the router is doing offloading of everything it is capable of.