Backup solutions in the age of 8TB consumer hard-disks

Forums › IT Pro and developers › Backup solutions in the age of 8TB consumer hard-disks

1 | 2 | 3

1043 posts

Uber Geek

#1733276 8-Mar-2017 20:23

I use FreeBSD and ZFS. It's great.

I know that somebody here is about to say that it is a crazy resource hog but that is just so not true.

You can run a 100TB setup on an old core 2 duo CPU with 2G of memory and saturate a Gigabit LAN with file transfers.

You will also hear people say that you must use ECC memory with ZFS. That just shows that those people simply fail to understand how good ZFS is and how bad everything else is. ECC is recommended by some folks on the basis that ZFS is such a secure way to store your data that the only realistic way that you could get data corruption is through a RAM error that would otherwise be picked up by ECC memory. All of the other ways to store your data have so many ways to lose it that you would worry about those long before you would worry about RAM errors. For example, a RAID 5 system using ECC memory is much more likely to lose your data than a ZFS system running the equivalent raidz1 without ECC.

That deals with the most common complaints that you will hear about a FreeBSD/ZFS solution but to move on, why is it so great?

1) You will never outgrow it. If you tried to build s storage system that was too big for ZFS you would run into other problems. Most likely, it would have so much mass that it would collapse into a black hole and you would never get your data out past the event horizon.

2) You can build a FreeBSD/ZFS system from standard consumer grade hardware. ZFS was designed that way. You do not need NAS optimized hard drives and you do not need hardware based RAID controllers. You do not need hard disk controllers with battery backups. That is a good thing in that you will save a lot of money on your hardware but consider what happens if you have a proprietary NAS appliance and the electronics dies. Your hard drives may be fine but if the proprietary RAID controller is toast and the company that made it is bust or has discontinued the item then your data is gone unless you can find the exact same replacement part and if that is the situation then you can bet that the price for the item will skyrocket.

3) FreeBSD/ZFS is the most reliable way to store your data at a single location. All of the features that you need for data integrity are there and they are well tested and they work. If you are worried about drive failures - and who isn't - then you can configure a FreeBSD/ZFS system to tolerate any number of drive failures that you wish.

4) FreeBSD/ZFS is flexible. I can add an extra drive to my system or replace one drive with a larger one and get access to the additional space by executing a few commands. To do this, you must have configured your system to support this from day one but if I can work out how to do it then why it be an issue for you?

5) FreeBSD/ZFS does your backups. If by backup you mean protection against you accidentally deleting a file or suffering from a rogue app that corrupts half of your PhD thesis or getting infected with ransomware that encrypts your wedding photos then snapshots - a built in feature of ZFS - will save the day. Snapshots are fast and use pretty well zero resources. If you take a snapshot today then, tomorrow, you can delete all of your files and they are all still there in the snapshot. You do not need to go buy an extra hard disk and spend time copying files over. Just type in the snapshot command you you are done. In an instant, you have locked down a snapshot of your entire file system. You can always refer back to that snapshot and get any of the files that were there at the time that you made the snapshot. Making a snapshot takes you pretty well no time and uses just a tiny amount of drive space. Given the reliability and robust nature of ZFS it is hard to see the point of buying extra hard disks to allow you store a local copy of a critical file. You cannot lose files in a snapshot - it is utterly read only you can only get rid of them by logging with superuser privs and removing the snapshot.

FreeBSD/ZFS does not provide offsite data storage - you need online backups or access to alternative locations. If anyone wants to know how I handle that then just ask.

timmmay

20587 posts

Uber Geek

Trusted

Lifetime subscriber

#1733316 8-Mar-2017 21:08

@jpoc, very interesting, but it doesn't really answer the question.

I lost a little data because of bad ram in my i7 system. Took ages to figure out, memtest x86 reported fine, another memory checker found the problem.

I considered building a ZFS system, but everyone said "use ECC". By the time I got a motherboard, Xeon, ECC RAM, etc, it was an expensive full computer. I considered low end, but all reports said performance wouldn't be great. In the end I used ReFS and Storage Spaces on my PC because it's on all the time anyway. ReFS with Storage Spaces is somewhat similar to ZFS, though much newer and I think simpler. It works fine on my i7 and two 4TB disks in a mirror configuration. So far, so good.

JimmyH

2886 posts

Uber Geek

#1733322 8-Mar-2017 21:27

My primary storage is a NAS box, in RAID5 (soon to be in RAID6) to mitigate against irritating hardware failure. However, I clearly understand that RAID is not a backup.

For day to day backups of my data directory, I just use a pair of USB3 external hard drives, and rotate them. I also have a couple more USB drives for longer-term storage of backup copies.

For stuff that won't change much (eg family photos/video, which are irreplaceable and which I really care about and I want to keep), I store them on external hard drives as per above. However, those drives can fail, so I tend to also burn two copies onto 50GB blu ray disks for cold storage. One set is stored in my garage (which is separate from the house), and the other at a family members house. Although burning them is tedious, you only have to do it once. They have a long archival life compared to hard drives, are pretty much immune to calamities such as being dropped, and are pretty cheap (I import spindles and they work out at circa $60-65 per TB). Plus, I take comfort in having a duplicate copy of each disk.

If you go the optical disk route make sure that you get HTL disks instead of the (slightly cheaper) LTH ones, as they are more reliable and a much better longevity. And don't buy them in NZ - good ones are hard to find (mostly you just find the crappy Verbatim LTH ones), and the pricing here is absurdly high.

For stuff you really care about you could go one better and get M-Disks (either Blu Ray or DVD), but these are pricey and not all drives will cope with them. In theory, these are claimed to have an archival life of 500-1000 years. I haven't tried these myself.

jpoc

1043 posts

Uber Geek

#1733921 9-Mar-2017 19:46

timmmay:

@jpoc, very interesting, but it doesn't really answer the question.

I lost a little data because of bad ram in my i7 system. Took ages to figure out, memtest x86 reported fine, another memory checker found the problem.

I considered building a ZFS system, but everyone said "use ECC". By the time I got a motherboard, Xeon, ECC RAM, etc, it was an expensive full computer. I considered low end, but all reports said performance wouldn't be great. In the end I used ReFS and Storage Spaces on my PC because it's on all the time anyway. ReFS with Storage Spaces is somewhat similar to ZFS, though much newer and I think simpler. It works fine on my i7 and two 4TB disks in a mirror configuration. So far, so good.

Your memory problem would of course affect any filing system but it is not what ECC memory is mainly for. Bad RAM might be bad but never show up as triggering the ECC and it should be detected by memory testing. ECC is there to give you protection from random individual bit-flips in ram that happen when something like a naughty muon from a cosmic ray impact passes through your memory chips. According to google's figures, a typical desktop PC will experience a bit flip as a one time event every few months. You can do nothing about that other than to use ECC memory. You might run a PC for years and never see bad ram but you will see cosmic ray bit flips several times a year.

Personally, I think that it makes sense to use ECC ram on all of my machines so I am migrating to AMD as they provide ECC on just about everything and do not have Intel's price gouging for it.

As for ReFS, my main objection is that it is proprietary. ZFS is open source and much less likely to contain bugs or back-doors. ReFS lacks many features and so far as I can tell, the only claimed advantage is that you do not need huge amounts of ram to run a system supporting deduplication. That is of course because it does not support it in the first case - and neither does it support a whole lot of other stuff.

You also suffer from the problem that Microsoft's file system & networking stacks support "corrupt on copy" which is a shocking thing to see on a modern file system. I became suspicious of this about 10 years ago and so I tried copying large numbers of large files from one windows box to another. Then I would run an MD5 checksum on both files. Once every few Terabytes, I'd see an error. It is worse when using usb disks but I have seen it on all drive hardware and all versions of windows since 2k.

I am still certain the FreeBSD/ZFS with snapshots does everything that you could want to do _in-the-home_ to cover reliability, resilience and backup and that, if you go that way, all you need is a good offsite strategy and you are setup for anything.

jpoc

1043 posts

Uber Geek

#1733961 9-Mar-2017 20:38

JimmyH:

...

For stuff you really care about you could go one better and get M-Disks (either Blu Ray or DVD), but these are pricey and not all drives will cope with them. In theory, these are claimed to have an archival life of 500-1000 years. I haven't tried these myself.

I have not tried M-disks and I have only ever burned a few hundred Blu-ray disks so I will not comment on longevity.

I have burned several thousand CD and DVD media over the years and I would not trust one for six months. I have seen plenty of those disks go unreadable in 12 months even with optimal storage.

There is a fundamental problem with all optical disks that I have seen.

They all have errors from day one, that is the way that they are. They also can all grow errors spontaneously over time, again that is the way they are.

They get around this problem by having error correcting data on the disk but there is a catch. Suppose that you burn a disk and decide to be extra safe and do a read-and-verify-after-burn as well. You might see one burn failure every twenty, fifty or hundred disks. That is OK, it's not so many and you can afford the cost and the time to chuck that disk and burn another copy. The next copy reports all is good so you are happy, but how happy should you be?

Your disk burning program just told you that it could read the disk and that the data matched the original. What it does not tell you is how much it had to rely on the error correction data on the burned disk in order to read back the data.

Your disk might be perfect or it might have needed every error correcting bit on the disk in order to read the thing back. If that was the case, you are just one bit fail away from an unreadable disk.

That is why I will not trust any optical disk.

jmorgannz

23 posts

Geek

#1734015 9-Mar-2017 21:29

Bluray disks are quite different to CD / DVD

Jase2985

13463 posts

Uber Geek

ID Verified

Lifetime subscriber

#1734162 10-Mar-2017 06:59

ive never had a CD fail out of the few hundred i have burnt, maybe im just lucky?

Quic Broadband

Move to New Zealand's best fibre broadband service (affiliate link). Free setup code: R587125ERQ6VE. Note that to use Quic Broadband you must be comfortable with configuring your own router.

timmmay

20587 posts

Uber Geek

Trusted

Lifetime subscriber

#1734172 10-Mar-2017 07:45

jpoc:

Your memory problem would of course affect any filing system but it is not what ECC memory is mainly for. Bad RAM might be bad but never show up as triggering the ECC and it should be detected by memory testing. ECC is there to give you protection from random individual bit-flips in ram that happen when something like a naughty muon from a cosmic ray impact passes through your memory chips. According to google's figures, a typical desktop PC will experience a bit flip as a one time event every few months. You can do nothing about that other than to use ECC memory. You might run a PC for years and never see bad ram but you will see cosmic ray bit flips several times a year.

Personally, I think that it makes sense to use ECC ram on all of my machines so I am migrating to AMD as they provide ECC on just about everything and do not have Intel's price gouging for it.

As for ReFS, my main objection is that it is proprietary. ZFS is open source and much less likely to contain bugs or back-doors. ReFS lacks many features and so far as I can tell, the only claimed advantage is that you do not need huge amounts of ram to run a system supporting deduplication. That is of course because it does not support it in the first case - and neither does it support a whole lot of other stuff.

You also suffer from the problem that Microsoft's file system & networking stacks support "corrupt on copy" which is a shocking thing to see on a modern file system. I became suspicious of this about 10 years ago and so I tried copying large numbers of large files from one windows box to another. Then I would run an MD5 checksum on both files. Once every few Terabytes, I'd see an error. It is worse when using usb disks but I have seen it on all drive hardware and all versions of windows since 2k.

I am still certain the FreeBSD/ZFS with snapshots does everything that you could want to do _in-the-home_ to cover reliability, resilience and backup and that, if you go that way, all you need is a good offsite strategy and you are setup for anything.

Interesting :) I've been meaning to change to ECC next time I update my computer, but the i7-2600K is going strong and I can't see any reason I'd upgrade it. I was thinking low end workstation Xeon chip, though I guess AMD could be an option with their new CPUs. I trust Intel more though.

The big advantage of ReFS is it's easy. I don't think W10 can access internal ZFS disks. ZFS would've required a custom build computer acting as a NAS, which costs money to build and run, and then even with a gigabit network it would reduce the disk access speed.

I use teracopy to validate checksums when copying files.

1101

3122 posts

Uber Geek

#1734229 10-Mar-2017 09:43

Jase2985:

ive never had a CD fail out of the few hundred i have burnt, maybe im just lucky?

Ive had plenty fail over the last 15years.
OK untill the day you need it. One day just unreadable.

Then again, archived tapes, tape drives and HD's also can fail.

richms

28191 posts

Uber Geek

Trusted

Lifetime subscriber

#1734255 10-Mar-2017 10:01

I found my old pile of ripped and burned copies of audio CDs and thought I would play some of them while my server full of flac files is stored away because renevations.

Most of them had at least one skip/stutter or random ticking noise at parts of it. Ranging from late 90's kodak gold discs from gamma computers when ihug was in newton road to dirt cheap spindles of 100 from random trademe sellers. Problems across the lot of them.

Richard rich.ms

JimmyH

2886 posts

Uber Geek

#1734511 10-Mar-2017 19:34

jpoc:

I have burned several thousand CD and DVD media over the years and I would not trust one for six months. I have seen plenty of those disks go unreadable in 12 months even with optimal storage.

There is a fundamental problem with all optical disks that I have seen.

That might be true for LTH disks which, like CDs and DVDs, use an organic dye that is prone to degrading in sunlight etc.

It's not true for HTL disks, which cost a bit more, but use a different chemistry and (apparently) have a much better storage life. That's why HTL disks are preferred over LTH for anything you want to keep.

I am, however, discovering that burning several terrabytes to optical disks is tedious - even in 50GB chunks (46.4GB actually).

nzkc

1572 posts

Uber Geek

#1734513 10-Mar-2017 19:55

If you start looking at online solutions, I'm a very happy Backblaze user. Saved my bacon a couple of times. Admittedly I dont have anywhere near 10TB. They charge you $50/yr for unlimited amount backed up.

If you choose them please use this invite (I'll get a free month and so will you): https://secure.backblaze.com/r/00n5el

jmorgannz

23 posts

Geek

#1734519 10-Mar-2017 20:26

I actually don't mind the idea of cloud backup - it all seems nice and convenient.
It's just the initial upload that's the killer.

I know Crashplan used to let you send in a seed drive for the first backup - but discontinued it.
I imagine other providers do it - but unlikely from NZ.

When I was looking at online backups, Backblaze was definitely one of my favorites.

On another track; I've managed to find somewhere that deals with off-lease server equipment willing to sell me an LTO4 external drive plus a SAS controller for ~$500

Sounds like it's going to suit me the best.

Does anyone else have issues logging in from their mobile?
If I try to login to reply on my mobile I get an incorrect username/password; but desktop works fine.

Rather strange; definitely tried retyping the login/password numerous times (easier than getting off the couch and going to PC to reply lol)

timmmay

20587 posts

Uber Geek

Trusted

Lifetime subscriber

#1734540 10-Mar-2017 21:22

BackBlaze does parallel uploads. CrashPlan has an Australian data center. Both are good.

BTR

1527 posts

Uber Geek

#1736392 14-Mar-2017 16:05

I raid 4 x 2TB disks in a Raid 10 in my Mac and back them up to a NAS device. I am also in the process of moving a copy of my photos to a cloud based photo album as well. If you have UFB crash plan or something similar might work for you.

1 | 2 | 3