Data integrity and md5 checksums failing

Forums › Desktop computing › Data integrity and md5 checksums failing

timmmay

20429 posts

Uber Geek

Trusted

Lifetime subscriber

#160085 23-Dec-2014 07:18

As mentioned in this thread I have quite a few backups, including offsite. I use FastSum command line version to do an MD5 of rarely changing data files on my internal disks, then when I backup to my external/offsite disk I run the checksums. Recently they've stated failing occasionally, I've found that when I run the checksum utility on the internal disk they usually also fail. Interestingly if I run the checksum utility twice I get slightly different readings - like one or two bytes have been read incorrectly. File types that changes have been detected on include jpeg, ghost, raw files.

I guess it's not unreasonable that when I read 5TB from disks a few bytes don't come off perfectly, but it's slightly worrying for bit rot. My disks seem fine, no SMART errors, which I know doesn't mean they're perfect or not failing.

Just interested in thoughts and opinions.

1 | 2 | 3

1508 posts

Uber Geek

#1202433 23-Dec-2014 08:30

If you are doing every file on the disk, surely that includes things like temp files and folders which may be written to after or during the copy. It may include event viewer files which are always changing, maybe just run the md5 tool over critical files or see if you can exclude changeable locations.

Try Vultr using this link and get us both some credit:

http://www.vultr.com/?ref=7033587-3B

AliExpress

Shop now on AliExpress (affiliate link).

timmmay

20429 posts

Uber Geek

Trusted

Lifetime subscriber

#1202435 23-Dec-2014 08:33

I only monitor data files that are relatively static. Examples of files that have changes are jpeg, raw, and ghost images.

SaltyNZ

8109 posts

Uber Geek

Trusted

2degrees

Lifetime subscriber

#1202506 23-Dec-2014 10:17

Backups are definitely a good idea, but you should look into using RAID with a filesystem like btrfs. Most filesystems assume that if the disk returned the requested info, then it's correct. But if you are finding that static files are occasionally returning conflicting results (and yes, this is much more common on modern high density platters) then a disk could have a block preparing to die. Btrfs checksums on a per-block level and checks it at read time. If the calculated checksum is wrong, it tries the redundant storage instead to get or rebuild the correct data. It also has a lot of other lovely features. Quite a few cheap NAS boxes support it these days. I have a NetGear ReadyNAS 312 with a pair of 3TB drives mirrored in it - it uses btrfs by default.

And of course as I write this, it occurs to me just to login to the box and check... And a disk has failed. Which means the email alerts on it aren't working. :-/ On the bright side I happen to know that the two disks have Seagate warranty until August next year because the other one failed about 2 months ago. Yay for mirroring!

tl;dr thanks timmmay for prompting me to check my NAS and find it had a failed disk it didn't alert me about.

iPad Pro 11" + iPhone 15 Pro Max + 2degrees 4tw!

These comments are my own and do not represent the opinions of 2degrees.

timmmay

20429 posts

Uber Geek

Trusted

Lifetime subscriber

#1202534 23-Dec-2014 10:36

A NAS with a good robust file system does sound like a good idea. Unfortunately at $1000 it's more than I can spend right now.

Can you use Btrfs with windows? Is there a similar system/scheme/format for windows? If I lose 20% of my disk capacity that's not a problem, I just buy a bigger disk, reliability is more important than space generally. Performance is moderately important, though I'd just keep data with high needs on NTFS.

Mark

1653 posts

Uber Geek

#1202568 23-Dec-2014 11:39

Running MD5 on top of your existing filesystem is a waste of effort to be honest, if MD5 detects the error it's too late, you need those checks done by the filesystem or underneath it.

It's amazing how many enterprise environments don't even think of bit rot and just assume the storage arrays handles it, RAID does it right ? Oh noes it don't! :-(

So running a good modern filesystem is a must if you want to make sure your TiB of data is still the same TiB of data from 6 months ago, ZFS, BTRFS and ReFS all have the capability to keep checking and rechecking, but you need to make sure the machine itself is up to spec as well (so a UPS and ECC RAM are essential options), and there is the side effect that the discs will be active nearly all the time adding up the power bill, cooling requirements and noise.

timmmay

20429 posts

Uber Geek

Trusted

Lifetime subscriber

#1202595 23-Dec-2014 12:14

It is a bunch of effort, it was really just a check of "are my files being faithfully stored and copied". The answer being no now I have to do something about it.

Can I format a disk with a good reliable file system that's accessible by Windows? Do I have to buy a NAS and put in RAID and this file system? Not sure I want to have to sell the NAS idea to my wife...

I won't go as far as ECC and UPS, that risk I am willing to take, but not disks randomly corrupting data.

kama

30 posts

Geek

#1202605 23-Dec-2014 12:31

Check your ram, eg with memtest/+.

timmmay

20429 posts

Uber Geek

Trusted

Lifetime subscriber

#1202606 23-Dec-2014 12:32

kama: Check your ram, eg with memtest/+.

I have done in the past, no issues, but I'll do it again just in case.

Mark

1653 posts

Uber Geek

#1202649 23-Dec-2014 14:05

timmmay: It is a bunch of effort, it was really just a check of "are my files being faithfully stored and copied". The answer being no now I have to do something about it.

Can I format a disk with a good reliable file system that's accessible by Windows? Do I have to buy a NAS and put in RAID and this file system? Not sure I want to have to sell the NAS idea to my wife...

I won't go as far as ECC and UPS, that risk I am willing to take, but not disks randomly corrupting data.

If you have Windows 8 you can tweak the registry to enable ReFS (Google knows all! :-)
Mac OS X you are SOL unless you want to faff around with Fuse, but performance is awful with that.

Usually a "NAS" is just a Linux box with a pretty front end interface, you could look at FreeNAS which is very good and can be put onto a spare PC, but if you want to use ZFS the requirements get heavy (minimum 8GB RAM and then roughly 1GB of RAM per TB of storage and also ECC HIGHLY recommended), worth it though, good bit of software!

Going ECC is important to a storage server, there is no point it putting data nice n safe to disk if it got corrupted in RAM :-)

Maybe your best bet is still multiple copies on cheap fat disks at remote locations ? There is only so much you can cater for with wife controlled budgets :-)

SaltyNZ

8109 posts

Uber Geek

Trusted

2degrees

Lifetime subscriber

#1202681 23-Dec-2014 14:50

Mark: if you want to use ZFS the requirements get heavy (minimum 8GB RAM and then roughly 1GB of RAM per TB of storage and also ECC HIGHLY recommended), worth it though, good bit of software!

RAM is cheap, but really the ZFS RAM requirements only get crazy if you want data deduplication. Oracle tend to recommend not to enable dedup on Solaris boxes, where ZFS is the default these days.

iPad Pro 11" + iPhone 15 Pro Max + 2degrees 4tw!

These comments are my own and do not represent the opinions of 2degrees.

timmmay

20429 posts

Uber Geek

Trusted

Lifetime subscriber

#1202701 23-Dec-2014 15:12

I have Windows 7 and unfortunately no spare computers lying around. I'd have to buy a NAS if I wanted a robust file system I guess :(

richms

27976 posts

Uber Geek

Trusted

Lifetime subscriber

#1202742 23-Dec-2014 15:56

If I find sfv failures I replace the file with a good copy. Windows will usually error when copying if something fails but I have had some files slip thru since I started making sfv files for my own content and keepingg the torrent for other stuff.

Richard rich.ms

michaelmurfy

meow
13191 posts

Uber Geek

Moderator

ID Verified

Trusted

Lifetime subscriber

#1202778 23-Dec-2014 17:05

You could get a cheap HP Microserver, shove a bucketload of RAM into it and install FreeNAS like I have done - ZFS is really good.

Michael Murphy | https://murfy.nz
Referral Links: Quic Broadband (use R122101E7CV7Q for free setup)

Are you happy with what you get from Geekzone? Please consider supporting us by subscribing.
Opinions are my own and not the views of my employer.

SaltyNZ

8109 posts

Uber Geek

Trusted

2degrees

Lifetime subscriber

#1202818 23-Dec-2014 18:19

michaelmurfy: You could get a cheap HP Microserver, shove a bucketload of RAM into it and install FreeNAS like I have done - ZFS is really good.

How much did that set you back overall (if you don't mind my asking)?

iPad Pro 11" + iPhone 15 Pro Max + 2degrees 4tw!

These comments are my own and do not represent the opinions of 2degrees.

timmmay

20429 posts

Uber Geek

Trusted

Lifetime subscriber

#1202819 23-Dec-2014 18:23

richms: If I find sfv failures I replace the file with a good copy. Windows will usually error when copying if something fails but I have had some files slip thru since I started making sfv files for my own content and keepingg the torrent for other stuff.

What software do you use?

michaelmurfy: You could get a cheap HP Microserver, shove a bucketload of RAM into it and install FreeNAS like I have done - ZFS is really good.

Cheap is probably a relative term. What would it cost for the server and enough RAM?

1 | 2 | 3