Data integrity and md5 checksums failing

Forums › Desktop computing › Data integrity and md5 checksums failing

1 | 2 | 3

1508 posts

Uber Geek
+1 received by user: 211

#1202889 23-Dec-2014 20:23

starting at $1000 ish without disks or much RAM.

Try Vultr using this link and get us both some credit:

http://www.vultr.com/?ref=7033587-3B

jpoc

1043 posts

Uber Geek
+1 received by user: 289

#1203970 26-Dec-2014 11:25

I use Freebsd and ZFS. I have done so for years and have never lost any data despite hardware failures, disk failures and power outages.

It is easy to setup and run - I export shares with samba.

The hardware requirements are low. Ignore all the exaggerated horror tales about gigs of RAM per terabyte of disk. I have an old core 2 duo running with 4G of memory and it is just fine. You can also ignore all the guff you get from the likes of PB tech who will try to upsell you from a Green drive to a Red one. ZFS works well on all drive types.

BTRFS is claimed to do all that ZFS can do and more but every time I looked, important features are still in beta. ZFS has been heavy duty industrial grade production ready for years.

All you would need to get going would be a cheap old desktop from trademe and the disks that you want to use. I ssh in to my server so no need for another monitor.

I chose freebsd rather than solaris because freebsd does gpt partitioning really well. You can give each gpt partition an arbitrary label and then access partitions for ZFS with the labels and you can forget about the weird and wacky /dev/sda5p3 stuff. That is also an argument for using Freebsd and not Linux for this job.

Be aware that windows can quietly lose data as it writes to a file. I tested this out with fastsum and a binfile-diff utility a few years back. Windows would happily copy a large file from one disk to another with no reported errors. Fsum would report different md5 sums and the bin diff would highlight that the destination file was the same length as the original but that the dest file contained a block of some small number of Kbytes that were set to zero! I no longer use windows to copy any files. I use either teracopy (free edition) or, over bad networks, I use a program that I wrote that will cope with network errors and outages. Tera copy does its own checksums and if I use my program, I then use fsum. I have seen these windows errors on w2k, XP and 7. I daresay that vista and 8 will be no better.

For backups, I do this:

Split the files to be backed up into four lots of equal size.

Create a rar file for each of those four lots. The rar file will be a multi volume one and it will have a small number of recovery volumes.

I will then use quickpar over all of the resultant files to generate 50% redundant recovery files.

I write all of the rar files for the first set onto disk 1 and and so on up to disk four and then I write the par2 files onto disks 5 and 6.

I can then stash the disks into 6 different offsite locations.

If I want to recover one file, I just need to fetch the one of those six disks onto which the original file was rar'ed from its location. If the disk has bad sectors, then the rar recovery volumes will deal with that. If the whole disk has failed then I need to get at least four of the remaining 5 disks and quickpar will restore a copy of the failed disk and I can extract the file that way.

jpoc

1043 posts

Uber Geek
+1 received by user: 289

#1203992 26-Dec-2014 12:18

Mark: Running MD5 on top of your existing filesystem is a waste of effort to be honest, if MD5 detects the error it's too late, you need those checks done by the filesystem or underneath it.

It's amazing how many enterprise environments don't even think of bit rot and just assume the storage arrays handles it, RAID does it right ? Oh noes it don't! :-(

So running a good modern filesystem is a must if you want to make sure your TiB of data is still the same TiB of data from 6 months ago, ZFS, BTRFS and ReFS all have the capability to keep checking and rechecking, but you need to make sure the machine itself is up to spec as well (so a UPS and ECC RAM are essential options), and there is the side effect that the discs will be active nearly all the time adding up the power bill, cooling requirements and noise.

ZFS has no need of a UPS. Raid 5 has a bug - the raid 5 write hole - which means that UPS is a must for raid 5 or raid 6 if you want to avoid data loss after a power outage. ZFS was written specifically to not have that weakness.

Mark

1653 posts

Uber Geek
+1 received by user: 555

#1204035 26-Dec-2014 13:53

jpoc:

ZFS has no need of a UPS. Raid 5 has a bug - the raid 5 write hole - which means that UPS is a must for raid 5 or raid 6 if you want to avoid data loss after a power outage. ZFS was written specifically to not have that weakness.

I don't agree about not needing a UPS ... data going into the server does not instantly get written to the ZIL it's in RAM for an eternity (to a computer) so having a UPS hooked up gives the server time to do a controlled destine/shutdown.

ZFS is good, but it's not the holy saviour of data storage :-) It's more than capable for home use but you do have to make sure it has its essential tools (UPS, ECC RAM etc) and it is resource hungry (which is why not too many consumer NAS implementations use it ... that and licensing probably) If you can't meet it's minimum requirements you may as well just build a simpler RAID-5 or RAID-6 setup.

timmmay

20859 posts

Uber Geek
+1 received by user: 5350

Trusted

Lifetime subscriber

#1204048 26-Dec-2014 14:21

The Mrs has agreed data integrity is a priority. Any more nas recommendations that use a good robust reliable file system? Price is significant, prefer low maintenance, device rather than pc most likely. Min 2 disks, maybe 4 would be nice.

timmmay

20859 posts

Uber Geek
+1 received by user: 5350

Trusted

Lifetime subscriber

#1204049 26-Dec-2014 14:22

And thanks to jpoc for the huge useful interesting post :) Sounds good but prefer simple, ie nas.

Mighty Ape

Shop now at Mighty Ape (affiliate link).

jpoc

1043 posts

Uber Geek
+1 received by user: 289

#1204054 26-Dec-2014 14:37

timmmay: And thanks to jpoc for the huge useful interesting post :) Sounds good but prefer simple, ie nas.

Then use FreeNAS - it is a FreeBSD/ZFS distro that is pre-configured to be a consumer NAS. It is well supported by community forums.

timmmay

20859 posts

Uber Geek
+1 received by user: 5350

Trusted

Lifetime subscriber

#1204090 26-Dec-2014 16:23

Sounds good but I don't want a big pc, something tiny and fit for purpose maybe easiest for me. I know you can get small pc's but a dedicated device maybe easier.

SaltyNZ

8866 posts

Uber Geek
+1 received by user: 9546

Trusted

2degrees

Lifetime subscriber

#1204111 26-Dec-2014 17:01

timmmay: Sounds good but I don't want a big pc, something tiny and fit for purpose maybe easiest for me. I know you can get small pc's but a dedicated device maybe easier.

I'm pretty happy with the ReadyNAS but I'm a little suspicious of both disks in it failing within a month of each other. That might be a bad batch (of disks), maybe? But it might be heat. The system seems to run relatively hot.

iPad Pro 11" + iPhone 15 Pro Max + 2degrees 4tw!

These comments are my own and do not represent the opinions of 2degrees.

jpoc

1043 posts

Uber Geek
+1 received by user: 289

#1204115 26-Dec-2014 17:09

SaltyNZ:
timmmay: Sounds good but I don't want a big pc, something tiny and fit for purpose maybe easiest for me. I know you can get small pc's but a dedicated device maybe easier.

I'm pretty happy with the ReadyNAS but I'm a little suspicious of both disks in it failing within a month of each other. That might be a bad batch (of disks), maybe? But it might be heat. The system seems to run relatively hot.

Yes, heat is a big issue in these things. small plastic cases with inadequate fans......

I have a couple of 6T drives and they run hot even in a big well vented case.

jpoc

1043 posts

Uber Geek
+1 received by user: 289

#1204134 26-Dec-2014 17:36

timmmay: Sounds good but I don't want a big pc, something tiny and fit for purpose maybe easiest for me. I know you can get small pc's but a dedicated device maybe easier.

In your case, I might go for something like this off trademe.

http://www.trademe.co.nz/computers/desktops/no-monitor/auction-825070257.htm

I believe that you can fit three drives in there - a boot drive and two data drives - and it will go up to 8G of RAM.

FreeNAS does not need an EFI BIOS to run with 6T disks.

Sure that is an old machine but I have never actually had an ancient Dell desktop fail on me. (Same for IBM/Lenovo.)

New World

Shop on-line at New World now for your groceries (affiliate link).

richms

29100 posts

Uber Geek
+1 received by user: 10215

Trusted

Lifetime subscriber

#1204145 26-Dec-2014 18:00

Old machines will use more power than a new machine doing the same things so those budget ex lease machines are often a false economy for always on applications.

Richard rich.ms