Now have UFB but FTP downloads are still very slow!

Forums › New Zealand Broadband › Now have UFB but FTP downloads are still very slow!

1 | 2

rlevis

347 posts

Ultimate Geek

#855116 13-Jul-2013 23:32

kiwirock: Is this LPFM/StationPlaylist Ross Levis?

I Gav, yep that's me.

I just resolved the problem! After googling a bit, disabling all "Large Send Offload (IPv4)" options in the NIC settings on the server, I'm now getting very good MB/s speeds. I wish I had done this a long time ago.

dwl

371 posts

Ultimate Geek

#855125 14-Jul-2013 00:02

kiwirock: Is this LPFM/StationPlaylist Ross Levis?

The latency between here in the US makes TCP think the capacity must be smaller than it is since it seems a little slowish given the latency to get acknowledgements back from the other end.

Opening more TCP streams helps to work around this since there's more capacity still there, just not less latency.

That's kind of the simplest way to put it.

The worst example and scenario to effect capacity based on latency is satellite Internet.

Cheers,
Gavin Stephens.

To expand this traffic analogy:

1) we send some payload in a small car (carrying 64kB of data) and it needs to wait for a reply from the other end before another car can be sent - for the delay to the US this 64kB window with typical round trip time gives a rate around 3Mbps

2) we can use multiple lanes provided the intermediate networks have enough capacity and we can then multiply the rate by the number of lanes (streams)

3) if we have access to a bigger car we can send much more data by using a bigger load (window sizes can easily get to 2+Mbps with windows scaling) so even with the high round trip times much faster single thread rates are possible - with UFB I have seen over 100Mbps from the US and others have seen much more than that - this is still single threaded

4) there are limits where the roads can only handle a total amount of traffic whether it be spread across multiple vehicles or bigger payload per vehicle

The problem is you really want to take 6 passengers at a time but the Ferrari might only have room for 1 and whatever car you have is stuck to the same speed (in this case limited by the speed of light which I understand is marginally faster than the Auckland motorways).

Great to hear that server settings have seen the speed rise.

rileypriddle

21 posts

Geek

#858159 17-Jul-2013 16:58

I am the same with upload speeds

Originally i was with Vodafone ADSL and would get 60KiB upload then i switched to Telecom Vdsl and got around 15-20KiB/s and now im on Fibre (30/10) with telecom and get 32ish Kib/s

Is this just Telecom throttling FTP?

Talkiet

4792 posts

Uber Geek

Trusted

#858187 17-Jul-2013 17:13

rileypriddle: I am the same with upload speeds

Originally i was with Vodafone ADSL and would get 60KiB upload then i switched to Telecom Vdsl and got around 15-20KiB/s and now im on Fibre (30/10) with telecom and get 32ish Kib/s

Is this just Telecom throttling FTP?

No.

Cheers - N
\

Please note all comments are from my own brain and don't necessarily represent the position or opinions of my employer, previous employers, colleagues, friends or pets.

dwl

371 posts

Ultimate Geek

#858253 17-Jul-2013 18:45

rileypriddle: I am the same with upload speeds
Originally i was with Vodafone ADSL and would get 60KiB upload then i switched to Telecom Vdsl and got around 15-20KiB/s and now im on Fibre (30/10) with telecom and get 32ish Kib/s
Is this just Telecom throttling FTP?

I assume that the upload rate on UFB is 32 kBps = 256 kbps (if it was 32 kbps I would be really worried) and that it is for high delay destinations (e.g. 200+ms). There could be several reasons for the reduction compared to other services and will need some checks to isolate the problem area.

1) What rate do you get for an uplink speedtest to say the Telecom speedtest server ?

2) If this local test is getting towards 10 Mbps (probably top out about 9 Mbps or 1.1 MBps due to overheads) then what is the delay and loss to the distant server (large sized pings) ?

3) Can you get good uploads to other FTP servers local and more distant ?

4) If the there isn't any loss to the server in question (e.g. over 100+ pings) and delay is not excessive then it might be TCP settings but as the default 64 kB window is good for around 3 Mbps to the US the TCP settings don't seem a likely cause and we probably need to look elsewhere, especially if the same settings and same endpoints have been working better on ADSL.

On this last point, it is remotely possible that there are burst effects resulting in drops on the UFB service that were absorbed and smoothed by the ADSL service (but I would hope this isn't happening). In simple terms, the upload can send say 64kB of data at a time (then a big pause) and if this is seen by UFB as a high peak rate (e.g. 64kB sent in 10ms can be seen as 64k*8*100= 51 Mbps for a short time) and if there is a fast acting policer you might get discards, hence TCP could die. I had understood that the UFB uplink wasn't so burst constrained but perhaps there is some silly small Excess Burst Size (EBS) setting active.

If you can run Wireshark during a transfer startup it would help to understand what is happening.

freitasm

BDFL - Memuneh
79266 posts

Uber Geek

Administrator

ID Verified

Trusted

Geekzone

Lifetime subscriber

#858257 17-Jul-2013 18:57

The OP posted the fix (which is indeed on his own server configuration) a few replies above yours.

dwl

371 posts

Ultimate Geek

#858279 17-Jul-2013 19:33

freitasm: The OP posted the fix (which is indeed on his own server configuration) a few replies above yours.

Hopefully that is an answer but as this was posted after a solution was posted I had assumed the problem was still there - I look forward to update.

The solution seemed to apply for downloads but it might also help here if applied on the sending (in this case uploading end) server.

AliExpress

Shop now on AliExpress (affiliate link).

rlevis

347 posts

Ultimate Geek

#863327 22-Jul-2013 03:23

Actually, my solution was short lived. The problem returned a couple of hours later. Simply resetting the network card by changing any setting causes the transfer speeds to go full speed. But a short time later, it goes back to the 313 kilobyte per second maximum speed. It doesn't appear to be related to the number of other connections or data transfers occurring on the server. It's a very peculiar issue.

I've had the server hosting company switch to a different network card which was installed in the server. The results were identical. They have tried investigating, but they get 100 Mbps when they try a download, which is the maximum.

If anyone is interested, here is the file I'm testing with. It's 1GB in size.
www.stationplaylist.com/SPLDVD.exe

IE says it's going to take around 1 hour to download. After reseting the NIC in the server, which is in fact hosted in Montreal Canada, I can then download the file in well under 10 minutes.

dwl

371 posts

Ultimate Geek

#863340 22-Jul-2013 08:08

rlevis: If anyone is interested, here is the file I'm testing with. It's 1GB in size.
www.stationplaylist.com/SPLDVD.exe

It looks to me like it is window size limited. I'm sorry I don't know why that would be but it seems to be server end. I have a VM in Montreal which I have just now tried for both back to NZ and direct from this test file.

Trying over an ADSL connection here that gets a file from my VM that was peaking at 5.8 Mbps (ADSL limit) we get a much lower speed from the test file from your server. My delay over ADSL is probably higher than for your test and I max out at 1.8 Mbps for the test file (i would expect 2-3 Mbps to a lower delay NZ destination). The following tcptrace result shows window size limited to 64kB (sorry about the formatting, I don't know who to insert non-proportional font for code - the tcptrace output has two columns, one for each direction):

--- www.stationplaylist.com ping statistics ---
100 packets transmitted, 100 received, 0% packet loss, time 20664ms
rtt min/avg/max/mdev = 272.423/319.276/359.925/23.873 ms, pipe 2

host b: mail.stationplaylist.com:80
a->b: b->a:
req 1323 ws/ts: Y/Y req 1323 ws/ts: N/Y
adv wind scale: 0 adv wind scale: 0
req sack: Y req sack: Y

max win adv: 65535 bytes max win adv: 65160 bytes
min win adv: 14600 bytes min win adv: 8192 bytes
zero win adv: 0 times zero win adv: 0 times
avg win adv: 65015 bytes avg win adv: 65145 bytes
initial window: 131 bytes initial window: 1448 bytes
initial window: 1 pkts initial window: 1 pkts

throughput: 4 Bps throughput: 173621 Bps

RTT samples: 2 RTT samples: 1986
RTT min: 305.5 ms RTT min: 0.0 ms
RTT max: 320.9 ms RTT max: 99.9 ms
RTT avg: 313.2 ms RTT avg: 1.4 ms

Now doing the same file download to my VM in Montreal which has lower (but still some metro and queuing) delay I get 63 Mbps. Note the window size is still limited to 64 kB:

--- www.stationplaylist.com ping statistics ---
100 packets transmitted, 100 received, 0% packet loss, time 19892ms
rtt min/avg/max/mdev = 0.897/1.073/4.113/0.491 ms

host f: mail.stationplaylist.com:80
e->f: f->e:
req 1323 ws/ts: Y/Y req 1323 ws/ts: N/Y
adv wind scale: 0 adv wind scale: 0
req sack: Y req sack: Y

max win adv: 65535 bytes max win adv: 65160 bytes
min win adv: 5840 bytes min win adv: 8192 bytes
zero win adv: 0 times zero win adv: 0 times
avg win adv: 65514 bytes avg win adv: 65159 bytes
initial window: 131 bytes initial window: 1448 bytes
initial window: 1 pkts initial window: 1 pkts

throughput: 8 Bps throughput: 7334534 Bps

RTT samples: 2 RTT samples: 40585
RTT min: 0.9 ms RTT min: 0.0 ms
RTT max: 20.2 ms RTT max: 33.4 ms
RTT avg: 10.6 ms RTT avg: 0.0 ms
RTT stdev: 0.0 ms RTT stdev: 0.4 ms

I think the "req 1323 ws/ts" = N request from the server needs addressing. The fact that your provider can get 100 Mbps locally isn't an adequate answer - unless their SLA limits performance to local traffic ;-) Hope this helps.

dwl

371 posts

Ultimate Geek

#863341 22-Jul-2013 08:16

Another perspective - I added 200 ms delay in Montreal on my VM interface which now gave this delay to your server:

--- www.stationplaylist.com ping statistics ---
100 packets transmitted, 100 received, 0% packet loss, time 38585ms
rtt min/avg/max/mdev = 200.960/201.067/202.150/0.617 ms, pipe 2

This has slowed what was a 63 Mbps transfer down to:
0% [ ] 6,638,740 312K/s eta 59m 50s

The only change was delay and when I then remove the emulated delay:
34% [==========> ] 384,796,316 7.41M/s eta 4m 15s

mercutio

1392 posts

Uber Geek

#863547 22-Jul-2013 13:24

probably is combination tcp window size and packet loss.

chicago:
# curl -O www.stationplaylist.com/SPLDVD.exe
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
6 1061M 6 73.2M 0 0 9.9M 0 0:01:46 0:00:07 0:01:39 10.1M^C

los angeles:
# curl -O www.stationplaylist.com/SPLDVD.exe
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
1 1061M 1 14.1M 0 0 1475k 0 0:12:16 0:00:09 0:12:07 1501k^C

nz:
# curl -O www.stationplaylist.com/SPLDVD.exe
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
1 1061M 1 11.0M 0 0 1087k 0 0:16:39 0:00:10 0:16:29 1361k^C

dwl

371 posts

Ultimate Geek

#863579 22-Jul-2013 13:51

mercutio: probably is combination tcp window size and packet loss.

These results do suggest packet loss as I reckon Chicago is only about 23 ms away and Los Angeles 75 ms so should scale up better compared to NZ at 200 ms.

I think some changes are being made at the server end as I now see window scaling working with an average window size of 7.4 MB when I use 200ms emulated delay with a stable speed of 98 Mbps.

Now I guess the challenge is to get the setup to be stable.

rlevis

347 posts

Ultimate Geek

#863641 22-Jul-2013 15:22

Currently it's working at full speed for some reason. I received this from the server company at 10am this morning...

"I have reviewed your network configuration and it was miss-configured. I have re-configured properly, with "auto negotiation on" on your server and with "100mbps full duplex" on the switch."

I had already changed the NIC setting between Auto Negotiation and 100Mbps full duplex several times without it permanently resolving it. So perhaps the switch configuration change has made some difference.

dwl

371 posts

Ultimate Geek

#863725 22-Jul-2013 16:29

rlevis: So perhaps the switch configuration change has made some difference.

I am pretty confident your issue has been window size with window scaling disabled. I struggle to see how interface speed negotiation by itself can affect this TCP setting but there may be other aspects of the interface profile that have also been adjusted and here is hoping that it is now stable.

1 | 2