Help! Perform error correction while downloading?

Forums › New Zealand Broadband › Help! Perform error correction while downloading?

tecumseh

3 posts

Wannabe Geek

#11689 6-Feb-2007 15:58

Every large file I download is corrupt, as per the thread here and also at gp.

http://www.gpforums.co.nz/thread/348250/?s=

Is there any download manager that slows the download to check for data corruption on the fly? Hash checks on 1MB sections or something, and then automatically re-downloads any corrupt sections?

I don't know if this is even possible...

But Torrent clients do this, so I can download torrents fine. If I could find a way to do this with normal downloads, I would be very happy.

Windows or Mac.

Cheers
Tec

barf

643 posts

Ultimate Geek

#59884 6-Feb-2007 17:38

this is symptom of ATM congestion/over-subscription, when 53 byte cells are lost (=~40 lost bytes to higher protocol layers?) TCP & PPP's CRC16 check won't always fail. Telecom should know about it already they will have alarms in place.

However, this is mere speculation. Hopefully I'll be proven wrong! (sbiddle, fraktul?)

Stick to using Bittorrent, emule or anything else that uses MD5 hash checks.

Sniffing the glue holding the Internet together

hellonearthisman

1819 posts

Uber Geek

Trusted

#59900 6-Feb-2007 20:12

Most download managers should do all that CRC checking on the packets it get sent.

You could try to reduce the number of splits that your download manager breaks the download into.
The more streams u have the more chance of them getting dropped/packet filtered/confusing telecoms stuff.

tecumseh

3 posts

Wannabe Geek

#59993 7-Feb-2007 13:01

barf: this is symptom of ATM congestion/over-subscription, when 53 byte cells are lost (=~40 lost bytes to higher protocol layers?) TCP & PPP's CRC16 check won't always fail. Telecom should know about it already they will have alarms in place.

However, this is mere speculation. Hopefully I'll be proven wrong! (sbiddle, fraktul?)

Thanks for the reply. The following bit of info about the cause of the corruption was posted elsewhere:

In short, Telecom have a problem with their Alcatal ASAM equipment, at least in the Ponsonby exchange and are working with Alcatel to find out what's going on. If you can get you line switched over to the Nokia DSLAM then your problems will go away.

I was able to compare a large test file on a host server with what was downloaded over my faulty connection while running an Ethernet capture on both sides (sending and receiving systems), and then identify the packets in error. At the TCP/IP level, the fault shows as a low-order bit swap within a TCP data packet, followed by the reverse low-order bit swap later in the same packet (some multiple of 128 bytes later). The TCP data packet only has a simple checksum on it, and hence this type of bit-swap error is not detected at the TCP/IP level. However the lower transport levels (PPPoA and ATM AAL5) do have CRC checks on their payload which is designed to catch these types of errors. So why are they being ignored? Why aren't the underlying protocols recovering from these bit errors? And have Telecom thought to set up their own test-case and watch the actual ATM cells??

This is why the normal CRC checks are not working, and why I'm looking for some kind of double-checking downloader, like torrent or emule clients do.