[TelstraClear] TelstraClear Cable connections appear to be rate shaped in one way or another

Forums › One NZ (including Farmside) › [TelstraClear] TelstraClear Cable connections appear to be rate shaped in one way or another

jnawk

176 posts

Master Geek
+1 received by user: 11

#89922 11-Sep-2011 14:54

Is anyone else having issues with individual connections appearing to be shaped? Myself and a few friends regularly transfer files to and from a machine in the USA, and those of us on Telstra Clear Cable regularly get a maximum of 3-4mbit/sec (with 15mbit/sec connections), but another friend using Xnet ADSL gets more like 10mbit/sec. I can get the full 15mbit/sec to the box, if I have multiple connections running at the same time - each gets 3-4mbit/sec.

1 | 2 | 3 | 4 | 5 | 6 | ... | 12

30853 posts

Uber Geek
+1 received by user: 9996

Retired Mod

Trusted

Biddle Corp

Lifetime subscriber

#519580 11-Sep-2011 15:35

You're presumably not aware of the speed limitations of TCP connections due to the distance from the US - because TCP has to send an ACK, it severly limits the maximum speed for a single connection. There will also be variances depending on your OS - if you're running XP the TCP window size is lower than newer OS's.

jnawk

176 posts

Master Geek
+1 received by user: 11

#519638 11-Sep-2011 18:24

I'm well aware of that kind of thing, and that doesn't explain why my friend, as mentioned in the original post, is perfectly capable of getting a much faster connection under exactly the same conditions except for the ISP is not Telstra Clear. The only thing I did not mention is that friend is also in New Zealand (same city).

In any case, TCP handles latent connections reasonably well by utilising such techniques as delayed ACK, slow start, and sliding windows.

michaeln

238 posts

Master Geek
+1 received by user: 44

Trusted

#519704 11-Sep-2011 21:51

You said to and from the overseas site. What are the 'from' rates you and your xNet friend is getting? What are the 'to' rates?

Can you share the name of the site, so we can see the route that might be taken?

Is it possible that the transfers might be cached?

Are you and your friends using identical operating systems? The RWIN setting varies dramatically between operating systems, and between different versions of operating systems.

jpollock

600 posts

Ultimate Geek
+1 received by user: 5

Trusted

#519767 12-Sep-2011 08:15

I've noticed that if it goes off of Telstra Clear's network, they have a limitation of around 3mbps per socket connection. Other ISPs don't have this limitation.

Before TCL got a steam mirror, it affected steam, and it affects all VPN connections (1 socket).

Jason

---
http://blog.jason.pollock.ca

jnawk

176 posts

Master Geek
+1 received by user: 11

#519776 12-Sep-2011 08:53

michaeln: You said to and from the overseas site. What are the 'from' rates you and your xNet friend is getting? What are the 'to' rates?

Can you share the name of the site, so we can see the route that might be taken?

Is it possible that the transfers might be cached?

Are you and your friends using identical operating systems? The RWIN setting varies dramatically between operating systems, and between different versions of operating systems.

File transfer over SSH connection from baird.net.nz. I get about 3mbit/connection, but can get my full 15mbit if I have enough connections going. He gets basically his whole connection worth of bandwidth down a single connection (something like 10mbit/sec).
Both operating systems are openSUSE 11.4.

As I have mentioned previously - the only difference is the internet connection itself.

timmmay

20858 posts

Uber Geek
+1 received by user: 5350

Trusted

Lifetime subscriber

#519804 12-Sep-2011 09:49

I always assumed this was either a limitation of TCP, or some kind of latency or setting in the network stopping things reaching the line speed for one connection. With high latency to the US TCP does slow down, and 3-4Mbps is usually fine. I wonder if it's deliberate, or just how their network works.

It's not difficult to open multiple connections, even for uploading you might be able to do it, so maybe it's best to just do that.

I'd be interested to find out why... someone call customer service and ask them... i'm sure their highly trained customer service team will know the answer, whether it's the TCP stack, rate limiting, latency, etc ;)

New World

Shop on-line at New World now for your groceries (affiliate link).

Beccara

1473 posts

Uber Geek
+1 received by user: 517

ID Verified

#519815 12-Sep-2011 10:04

jnawk:
michaeln: You said to and from the overseas site. What are the 'from' rates you and your xNet friend is getting? What are the 'to' rates?

Can you share the name of the site, so we can see the route that might be taken?

Is it possible that the transfers might be cached?

Are you and your friends using identical operating systems? The RWIN setting varies dramatically between operating systems, and between different versions of operating systems.

File transfer over SSH connection from baird.net.nz. I get about 3mbit/connection, but can get my full 15mbit if I have enough connections going. He gets basically his whole connection worth of bandwidth down a single connection (something like 10mbit/sec).
Both operating systems are openSUSE 11.4.

As I have mentioned previously - the only difference is the internet connection itself.

Are you sure he's not doing anything different? He must be because that host is at best 160ms away from us which puts max TCP throughput at around 3mbit per TCP stream, For him to get 10mbit from a SINGLE TCP stream he would need to be within 50ms latency of that host

Most problems are the result of previous solutions...

All comment's I make are my own personal opinion and do not in any way, shape or form reflect the views of current or former employers unless specifically stated

jpollock

600 posts

Ultimate Geek
+1 received by user: 5

Trusted

#519835 12-Sep-2011 10:39

Huh, you learn something every day.

http://en.wikipedia.org/wiki/TCP_tuning

Says that the maximum transfer rate is:

throughput <= RWIN/RTT

Where:
RWIN is the receive window size
RTT is the round trip time.

The RWIN has a maximum value of 65536, so that means that given a RTT of 150ms, we have a throughput of:

(65536 Bytes/0.150s = 3,495,253.33 bits/s = 3.5mbps.

Since we can't make the window size any larger, the only thing we can change is the RTT.

Bummer.

There may be a way around it, http://en.wikipedia.org/wiki/TCP_window_scale_option . However, since we're going to the proxy, it will be scrubbing any HTTP connections we use, making any local options pointless anyways.

---
http://blog.jason.pollock.ca

jnawk

176 posts

Master Geek
+1 received by user: 11

#519857 12-Sep-2011 11:25

jpollock: Huh, you learn something every day.

http://en.wikipedia.org/wiki/TCP_tuning

Says that the maximum transfer rate is:

throughput
Where:
RWIN is the receive window size
RTT is the round trip time.

The RWIN has a maximum value of 65536, so that means that given a RTT of 150ms, we have a throughput of:

(65536 Bytes/0.150s = 3,495,253.33 bits/s = 3.5mbps.

Since we can't make the window size any larger, the only thing we can change is the RTT.

Bummer.

There may be a way around it, http://en.wikipedia.org/wiki/TCP_window_scale_option . However, since we're going to the proxy, it will be scrubbing any HTTP connections we use, making any local options pointless anyways.

Actually, in my case, as it is an SSH session (ie, encrypted), it is uncacheable. No proxy or anything is involved. And, I have control of both boxes, so I can increase the RWIN and the WWIN.

My observations are that the speed readings jump about a bit (I've probably not set the size optimally - 512k - probably a bit big), but TCP eventually settles down to the same old speed it always did.

Interestingly enough, my friend has left his RWIN at the default (128k), and he gets 1.1MB/sec or about 9mbit/sec - about 3 times the speed I get. His ping is 10msec better than mine (150ms vs 160ms)

Which leaves the only other option - Telstra Clear messing with the connection in some way - shaping, or futzing with the window values directly. I suspect shaping.

Beccara

1473 posts

Uber Geek
+1 received by user: 517

ID Verified

#519862 12-Sep-2011 11:31

Even at a 128k RWIN and 150ms he's still maxing out at 6-7mbit so something different is going on

jnawk

176 posts

Master Geek
+1 received by user: 11

#519877 12-Sep-2011 11:54

Beccara: Even at a 128k RWIN and 150ms he's still maxing out at 6-7mbit so something different is going on

Well, even if we assume he is getting 7mbit/sec, my RWIN was also initially at 128k, and as stated above - ping of 160msec, so that is still 6.5mbit/sec, which is nearly double the 3.3mbit/sec that I see.

Lego

Shop now for Lego sets and other gifts (affiliate link).

jpollock

600 posts

Ultimate Geek
+1 received by user: 5

Trusted

#519908 12-Sep-2011 12:56

Has anyone tried the M-Lab toolset to see what's going on?

http://www.measurementlab.net

I'll give them a try when I get home tonight.

Jason

---
http://blog.jason.pollock.ca

jnawk

176 posts

Master Geek
+1 received by user: 11

#519924 12-Sep-2011 13:18

Well, I've well and truly messed about with both stacks. (Mine & the server's). I got 3.6mbit/sec max. I'm not even sure if that was a result of my tweaking. Friend's connection still gets him 8-9mbit/sec, and his RWIN is still 128k.

Even if I had done no tuning, my RWIN was 128k also, and its now big enough for a 20mbit connection, yet my speed still settles on largely the same as it was.

Maybe it is time for this:

timmmay: I'd be interested to find out why... someone call customer service and ask them... i'm sure their highly trained customer service team will know the answer, whether it's the TCP stack, rate limiting, latency, etc ;)

jpollock

600 posts

Ultimate Geek
+1 received by user: 5

Trusted

#519945 12-Sep-2011 13:35

If you've got the ability to run a test and see what arrives on the server, try this:

I'm guessing they're stripping out the TCP extensions, since that's the only way to get a window size larger than 64k. So, let's check if it's getting to the server:

I'm assuming you're using linux here.

1) run tcpdump on your local machine, limit it to only packets going to the server.
2) run tcpdump on the server, limit it to only packets from your client.

Run your speed test.

Check the SYN packets for differences. We're looking for the TCP extensions, specifically the window scaling option.

I'm betting that TCL is stripping the extensions as a cheap way of rate limiting the connection, but we'll need to catch them at it before we can raise a ticket.

Jason

---
http://blog.jason.pollock.ca

jnawk

176 posts

Master Geek
+1 received by user: 11

#519978 12-Sep-2011 14:23

jpollock: If you've got the ability to run a test and see what arrives on the server, try this:

I'm guessing they're stripping out the TCP extensions, since that's the only way to get a window size larger than 64k. So, let's check if it's getting to the server:

I'm assuming you're using linux here.

1) run tcpdump on your local machine, limit it to only packets going to the server.
2) run tcpdump on the server, limit it to only packets from your client.

Run your speed test.

Check the SYN packets for differences. We're looking for the TCP extensions, specifically the window scaling option.

I'm betting that TCL is stripping the extensions as a cheap way of rate limiting the connection, but we'll need to catch them at it before we can raise a ticket.

Jason

14:14:14.764499 IP a.b.c.d.32768 > p.q.r.s.xyz:
Flags [S], seq 733879383, win 4380, options [mss 1460,sackOK,TS val 282915597 ecr 0,nop,wscale 8], length 0

02:14:08.765236 IP a.b.c.d.32768 > p.q.r.s.xyz:
Flags [S], seq 733879383, win 4380, options [mss 1460,sackOK,TS val 282915597 ecr 0,nop,wscale 8], length 0

Unfortunately, nope, they aren't doing that.

And yes, a.b.c.d, p.q.r.s, and xyz are manifestations of my tin-hat.

1 | 2 | 3 | 4 | 5 | 6 | ... | 12