rclone and dropped packets

Forums › LAN (ethernet/Wifi/routers/Bluetooth) › rclone and dropped packets

tek

112 posts

Master Geek

ID Verified

#273194 10-Aug-2020 07:25

Hello everyone!

So, I have a system where I use rclone to transfer data into Google Drive (I'm sure most know what I'm doing here).
My problem is that i'm experiencing about 15% packet loss on my WAN while this transfer is happening. Does not matter if I'm transfering at 50MB/s or 1MB/s. Whenever I transfer data using other services via a browser I do not experience any packet loss. Is there something that I may have missed in the rclone config or could my ISP be doing something to this type of traffic? I have thought about changing my router as a test but thats quite a process for the setup I have as it would involve a lot of config being copied across.

Could this be a limit of the USG not doing well with the TCP traffic generated from rclone?

My setup:

Router: UniFi USG (DPI enabled, IPS disabled)
ISP: Voyager
Server: unraid with rclone mounting my Google Drive as a seperate disk
All connectivity wired and have connected server direct to USG to no change

Image from my router showing the dropped packets during an rclone transfer and then antoher transfer later in the day with no loss.

BarTender

3585 posts

Uber Geek

ID Verified

Trusted

Lifetime subscriber

#2537563 10-Aug-2020 08:00

Are you shaping data upstream as if it’s trying to send at full speed then the LFC access network will drop packets if you try and burst faster than 500Mbit.

Tradepub

Free professional, reference and technical white papers (affiliate link).

tek

112 posts

Master Geek

ID Verified

#2537566 10-Aug-2020 08:08

BarTender: Are you shaping data upstream as if it’s trying to send at full speed then the LFC access network will drop packets if you try and burst faster than 500Mbit.

Not doing any shaping as I've never had an issue saturating my upload/downlaod using other methods. I don't have a tonne of knowledge on rclone but i've heard it can be much more aggressive. Is this correct?

SirHumphreyAppleby

2816 posts

Uber Geek

#2537619 10-Aug-2020 08:41

tek:

Not doing any shaping as I've never had an issue saturating my upload/downlaod using other methods. I don't have a tonne of knowledge on rclone but i've heard it can be much more aggressive. Is this correct?

Rclone is fairly persistent and needs to be given the various API and data limits imposed by by back-end providers. In most cases it will just be reusing connections, so TCP should take care of any congestion. It also backs off with incrementing delays when the back-end won't allow it to send/query data.

fe31nz

1189 posts

Uber Geek

#2538337 11-Aug-2020 00:57

Is this a fibre connection? What speed?

That graph shows your router dropping the packets. I would have expected any packet dropping to be being done in the upstream routers, not in yours. The way routers work is that when you send a packet, it goes on a transmit queue, waiting until it can be sent. If it gets sent, that is good. But if more packets get added to the queue and the queue overflows, packets need to be dropped. The router will have an algorithm for that, and it may drop the new packet, or it may drop an older packet to make room for the newer one, depending on the priority of the packets. I do not know about USGs, but in my EdgeRouter there are settings for which algorithm is used and what parameters are set for the algorithm. In any case, it is a sign of congestion - you are transmitting more than the available bandwidth out of your router to the next hop router at your ISP. If you are on a fast fibre connection, you normally want your router to not use any significant queuing algorithm as doing that will require the packets to all be routed using the CPU, which will dramatically reduce the maximum throughput. For fast fibre, you normally want to turn off any special queuing options and just let the hardware offloading do the routing. I think that with only hardware routing, it will simply drop any packet that arrives when the transmit queue is full, without regard to what sort of packet it was or its priority. If you have an ADSL or VDSL connection or a slow fibre connection, then you have the option to use an algorithm for the queue as the CPU can do the routing using the queuing algorithm faster than your connection speed. It is a while since I last looked at this as I have a gigabit fibre now, but I think that the recommendations are to use FQ-CODEL if you do need to control the queuing. Older algorithms cause a different problem called "buffer bloat". EdgeRouters do FQ-CODEL, so I would hope USGs do also. But tweaking your queuing settings is something that experienced network engineers are needed for - it is a pretty time consuming thing to learn enough about it to do it properly.

For TCP connections, getting a dropped packet tells TCP to back off the transmit rate, so it should only take a few dropped packets for TCP to have adjusted to the actual bandwidth available to you for the end-to-end TCP connection, and then it will transmit at that rate unless conditions change. Does rclone use lots of connections, or just one TCP connection? If it uses lots of connections, that could be the problem as the connections could be fighting each other for bandwidth for a while before they settle down. Also, Windows is pretty bad at doing this sort of TCP optimisation - if you are running rclone from Windows, it is likely to be a Windows problem. TCP works much better on Linux. On Windows, I have seen big downloads where the speed is as fast as the connection allows, then all of a sudden the speed drops rapidly to almost nothing, then a number of seconds later it will go up to full speed again. It keeps doing that for the entire download. Doing the same download on my Ubuntu box using the same software sees a steady high speed. On Windows, if rclone is configurable, it would be a good idea to try using just one upload connection. Then it will take a while (a minute or two) before Windows will have that connection running at full speed, but it should be able to sustain that speed unless something else starts needing lots of your bandwidth.

tek

112 posts

Master Geek

ID Verified

#2538538 11-Aug-2020 12:05

fe31nz:

Is this a fibre connection? What speed?

That graph shows your router dropping the packets. I would have expected any packet dropping to be being done in the upstream routers, not in yours. The way routers work is that when you send a packet, it goes on a transmit queue, waiting until it can be sent. If it gets sent, that is good. But if more packets get added to the queue and the queue overflows, packets need to be dropped. The router will have an algorithm for that, and it may drop the new packet, or it may drop an older packet to make room for the newer one, depending on the priority of the packets. I do not know about USGs, but in my EdgeRouter there are settings for which algorithm is used and what parameters are set for the algorithm. In any case, it is a sign of congestion - you are transmitting more than the available bandwidth out of your router to the next hop router at your ISP. If you are on a fast fibre connection, you normally want your router to not use any significant queuing algorithm as doing that will require the packets to all be routed using the CPU, which will dramatically reduce the maximum throughput. For fast fibre, you normally want to turn off any special queuing options and just let the hardware offloading do the routing. I think that with only hardware routing, it will simply drop any packet that arrives when the transmit queue is full, without regard to what sort of packet it was or its priority. If you have an ADSL or VDSL connection or a slow fibre connection, then you have the option to use an algorithm for the queue as the CPU can do the routing using the queuing algorithm faster than your connection speed. It is a while since I last looked at this as I have a gigabit fibre now, but I think that the recommendations are to use FQ-CODEL if you do need to control the queuing. Older algorithms cause a different problem called "buffer bloat". EdgeRouters do FQ-CODEL, so I would hope USGs do also. But tweaking your queuing settings is something that experienced network engineers are needed for - it is a pretty time consuming thing to learn enough about it to do it properly.

For TCP connections, getting a dropped packet tells TCP to back off the transmit rate, so it should only take a few dropped packets for TCP to have adjusted to the actual bandwidth available to you for the end-to-end TCP connection, and then it will transmit at that rate unless conditions change. Does rclone use lots of connections, or just one TCP connection? If it uses lots of connections, that could be the problem as the connections could be fighting each other for bandwidth for a while before they settle down. Also, Windows is pretty bad at doing this sort of TCP optimisation - if you are running rclone from Windows, it is likely to be a Windows problem. TCP works much better on Linux. On Windows, I have seen big downloads where the speed is as fast as the connection allows, then all of a sudden the speed drops rapidly to almost nothing, then a number of seconds later it will go up to full speed again. It keeps doing that for the entire download. Doing the same download on my Ubuntu box using the same software sees a steady high speed. On Windows, if rclone is configurable, it would be a good idea to try using just one upload connection. Then it will take a while (a minute or two) before Windows will have that connection running at full speed, but it should be able to sustain that speed unless something else starts needing lots of your bandwidth.

Thanks for this explanation! I will try changing the number of threads soon and report back.

I have a gigabit fibre connection rom Voyager and have no queueing enabled on the USG. I have actually also now tested the connection using a Draytek Vigor 2860 and I'm having the same issues.
I am using a Docker container on my unraid server to move data via rclone, so all linux.

Below are screenshots showing how I get 10-25% packet loss when the transfers are taking place. And they are only using a small portion of my available upload capacity.

Transfers Paused:

Transfers Resumed:

mdf

3486 posts

Uber Geek

Trusted

#2538565 11-Aug-2020 13:19

Might be worth asking on the rclone forums too. I've always found them pretty responsive.

fe31nz

1189 posts

Uber Geek

#2539020 12-Aug-2020 02:47

tek:

Thanks for this explanation! I will try changing the number of threads soon and report back.

I have a gigabit fibre connection rom Voyager and have no queueing enabled on the USG. I have actually also now tested the connection using a Draytek Vigor 2860 and I'm having the same issues.
I am using a Docker container on my unraid server to move data via rclone, so all linux.

Below are screenshots showing how I get 10-25% packet loss when the transfers are taking place. And they are only using a small portion of my available upload capacity.

The rclone data may only be taking a small portion of your 1 gigabit connection to Voyager, but the available bandwidth to your Google Drive server will be much less than 1 gigabit. Generally you will only get anything close to 1 gigabit to servers in New Zealand. Any overseas destination involves contention with all the other traffic from your ISP and others over that path. No ISP is going to have enough bandwidth to allow you to do 1 gigabit/s to overseas destinations! Let alone the bandwidth contention in the overseas routers. But the lost packets for the overseas connections should be happening in the routers where that contention is, not in your router. When you see packet loss testing with opsgenie, do you know what it is actually doing to generate those results? If it is sending ping packets to an overseas server, then there are several things that can be going on. Ping packets are often considered low priority by routers and are preferentially dropped. So when a router is getting busy, it may well still be able to handle all its real traffic, but be dropping most ping packets. Whatever traffic opsgenie is sending to do that test will be potentially in contention with your real traffic, so if it is sending a lot of data, that may be what pushes things over the edge and causes packet loss. The proper way to look for packet loss is either to get the router that is dropping packets to tell you about it, or to get your connections to tell you. TCP connections know when they are seeing packet loss due to the fact that they expect to receive an acknowledgment of receipt from the other end for all packets. So they can tell how many packets they send that the other end did not receive and they had to send again. Not all other protocols will have the same ability - some are specifically designed to have redundant data enough to allow for a certain level of lost packets and do not have any resend mechanism. But as you are using TCP connections from rclone, can rclone tell you about the packet loss somehow? That would be the definitive data about it. If you are good with Wireshark (or tshark or tcpdump) and can set up a filter to see TCP retransmissions, that is another way to see what is really going on if rclone can not tell you.

And one other thing to think about - how fast is the hard drive you are cloning the data from? If it is capable of only less than 1 gigabit/s (as many older drives are), then that will be the ultimate limiting factor for the rclone speed. If you have a fast SSD, try transferring data from the drive you are cloning to the SSD and see how fast that is.

K8Toledo

1009 posts

Uber Geek

#2539022 12-Aug-2020 03:39

tek:

Hello everyone!

So, I have a system where I use rclone to transfer data into Google Drive (I'm sure most know what I'm doing here).
My problem is that i'm experiencing about 15% packet loss on my WAN while this transfer is happening. Does not matter if I'm transfering at 50MB/s or 1MB/s. Whenever I transfer data using other services via a browser I do not experience any packet loss. Is there something that I may have missed in the rclone config or could my ISP be doing something to this type of traffic? I have thought about changing my router as a test but thats quite a process for the setup I have as it would involve a lot of config being copied across.

Could this be a limit of the USG not doing well with the TCP traffic generated from rclone?

My setup:

Router: UniFi USG (DPI enabled, IPS disabled)
ISP: Voyager
Server: unraid with rclone mounting my Google Drive as a seperate disk
All connectivity wired and have connected server direct to USG to no change

Image from my router showing the dropped packets during an rclone transfer and then antoher transfer later in the day with no loss.

Those packets weren't dropped they're being discarded.

It's probably due to low memory.

tek

112 posts

Master Geek

ID Verified

#2541452 15-Aug-2020 09:30

K8Toledo:

Those packets weren't dropped they're being discarded.

It's probably due to low memory.

Defiantly not low memory. My server and router are sitting below 40% used while a copy is in progress. I am still waiting for Voyager to get back to me to confirm if the connection is at fault as we only noticed it when moving providers.

K8Toledo

1009 posts

Uber Geek

#2541529 15-Aug-2020 12:31

tek:

K8Toledo:

Those packets weren't dropped they're being discarded.

It's probably due to low memory.

Defiantly not low memory. My server and router are sitting below 40% used while a copy is in progress. I am still waiting for Voyager to get back to me to confirm if the connection is at fault as we only noticed it when moving providers.

Certainly looks that way (could be low buffer in router but I suspect main memory).

When you say below 40% what are you referring to - Commit Charge in Task Manager or...? What does the Standby & Free list look like?

How much RAM installed?