Data integrity and md5 checksums failing

Forums › Desktop computing › Data integrity and md5 checksums failing

1 | 2 | 3

timmmay

20580 posts

Uber Geek

Trusted

Lifetime subscriber

#1208931 6-Jan-2015 08:20

I've looked at NAS devices, including FreeNAS. It seems I'd have to build a $700+ enterprise grade PC (not including drives) with min 8GB and ideally 16GB ECC RAM just to use FreeNAS, which seems like overkill. To get it to $700 I'd have to import from the states, building in NZ would cost $1500.

I'm considering using ReFS and Storage Spaces instead, a three disk parity set of say 3x4TB drives. That would provide bit rot detection and repair. I'm on W7 so I'd have to either buy Win 8.1 or use the free Win 10 tech preview - some risk going with a tech preview but buying 8.1 now would seem a bit silly given W10 is out in 3-6 months. Disk performance is meant to be a bit rubbish when writing, but reading is meant to be ok. Anyone have any thoughts or experience with ReFS/Storage Spaces?

Hammerer

2476 posts

Uber Geek

Lifetime subscriber

#1209967 7-Jan-2015 15:48

timmmay: I've looked at NAS devices, including FreeNAS. It seems I'd have to build a $700+ enterprise grade PC (not including drives) with min 8GB and ideally 16GB ECC RAM just to use FreeNAS, which seems like overkill. To get it to $700 I'd have to import from the states, building in NZ would cost $1500.

I'm considering using ReFS and Storage Spaces instead, a three disk parity set of say 3x4TB drives. That would provide bit rot detection and repair. I'm on W7 so I'd have to either buy Win 8.1 or use the free Win 10 tech preview - some risk going with a tech preview but buying 8.1 now would seem a bit silly given W10 is out in 3-6 months. Disk performance is meant to be a bit rubbish when writing, but reading is meant to be ok. Anyone have any thoughts or experience with ReFS/Storage Spaces?

I think that you might be confusing your options.

If data integrity is the primary requirement then why would you sidestep ZFS? You are talking about using other technology which generally delivers less, is less proven, is less open, and I suspect will cost as much if not more in the long run.

I don't think that you even need 8GB RAM let alone 16GB. That would be the case to start with and possibly longer term. If that is correct then your options become a lot cheaper.

You don't appear to need some features that increase the RAM ceiling.
From memory, you have said elsewhere in one of these threads that your current space requirement is relatively small and 4GB would be sufficient for your main files. Why not start with that so you need a lot less RAM.
It looks like you don't require high performance because it sounds like most of your files do not change or do so infrequently. Write performance, as you are aware, is more of a bottleneck but you probably don't have lots of concurrent users or concurrent updates.

Others have pointed out that the FreeBSD/FreeNAS hardware requirements depend on which features you use in ZFS. If you search for more info you will see that there is a large performance trade-off but that probably won't matter to you. It didn't matter to me so I had no problems with one quarter of the recommended RAM for ZFS with NAS4Free and the original FreeNAS, and more recently one half the RAM with FreeNAS.

It is easy enough to test it for yourself and monitor ZFS's memory usage while under the sort of loads you might have.

timmmay

20580 posts

Uber Geek

Trusted

Lifetime subscriber

#1209980 7-Jan-2015 16:05

ReFS is definitely less proven than ZFS, but provides roughly the same capabilities, and is production ready. My requirement is protected storage, not a NAS, so using the hardware I already have saves money. Given I will have a backup the data is safe even if it fails completely. FreeNAS is unsupported below 8GB, it has been made clear on their forums it really is a minimum even for a low usage home system - sounds like bad coding to me. Nas4Free apparently does have a much lower memory overhead.

To me ReFS and Storage Spaces eliminate one computer ($800+) and the associated power bill, with a minor loss in flexibility. That saving is more important to me than the minor flexibility and potential reliability gains at this point.

Thanks for your thoughts though, always good to get alternate viewpoints.

Hammerer

2476 posts

Uber Geek

Lifetime subscriber

#1210029 7-Jan-2015 17:01

timmmay: ReFS is definitely less proven than ZFS, but provides roughly the same capabilities, and is production ready. My requirement is protected storage, not a NAS, so using the hardware I already have saves money. Given I will have a backup the data is safe even if it fails completely. FreeNAS is unsupported below 8GB, it has been made clear on their forums it really is a minimum even for a low usage home system - sounds like bad coding to me. Nas4Free apparently does have a much lower memory overhead.

To me ReFS and Storage Spaces eliminate one computer ($800+) and the associated power bill, with a minor loss in flexibility. That saving is more important to me than the minor flexibility and potential reliability gains at this point.

Thanks for your thoughts though, always good to get alternate viewpoints.

I understand. I am in a similar situation having data integrity issues with few options to resolve it due to other priorities for our money. That's why I prefer the low overhead in NAS4Free. Incidentally, it is my three sons who are killing any chance of diverting money to a ZFS NAS because they are using about 400kWh of electricity a month for their computers.

Whatever you decide I'll be interested to hear how it works out for you.

timmmay

20580 posts

Uber Geek

Trusted

Lifetime subscriber

#1210042 7-Jan-2015 17:24

400kwh isn't so much if it's your whole power bill. With just my wife and I we use around 800kwh per month - total not just for computers.

I have no doubt storage spaces will be fine otherwise I wouldn't go for it. Means reformatting computer though, PITA, but I'll do a macrium backup. First disk is on the way, split the order to save GST, should get them both in a few weeks and have a shot.

Hammerer

2476 posts

Uber Geek

Lifetime subscriber

#1210421 8-Jan-2015 10:52

400kWh is just for their computers. We're currently using 1,200kWh a month when we normally use 700-900kWh at this time of the year. I could run and pay for a pretty good NAS for that.

1 | 2 | 3