Backup solutions in the age of 8TB consumer hard-disks

Forums › IT Pro and developers › Backup solutions in the age of 8TB consumer hard-disks

jmorgannz

23 posts

Geek

#208977 7-Mar-2017 13:41

Hi All,

A few years ago I decided I was never going to have a (personal) data loss issue again, and purchased a NAS for my home.

I have upgraded it a couple of times over the years, and currently it has 10TB of storage on it.
(2x 4TB RAID 1, 2x 6TB RAID 1)

This has given me a certain piece of mind with regard to hardware failure; but as is not reiterated enough, redundancy is not backup.

This has been bugging me of late, especially because if I accidentally delete something on the NAS drives, or some other unforeseen manipulation of the data, my redundancy is not going to help me one bit.

This leads me to the following question: What's a decent backup solution in the age of such large disks.

I have looked into online backups; but being capped at 100GB a month, it would take 100 months of full cap usage to back up all my data.
The only other thing that comes to mind is using tape-backup.

I am OK with the data being on 'slow storage', and the tape solution appeals to me as AFAIK they are designed for 'long term data storage', so perhaps may suffer less from bit-rot than other solutions.

The price, though, is not very appealing; especially if I were trying to minimise tape usage with something like LTO6, the drives themselves costing 3k, not to mention the tapes.
These are usually geared at more server type environments too; requiring a SAS connection.

So IT pro's, what's a geek to do about backup in the age where it's totally realistic to have ten's of terabytes of personal storage ?

Is there some magical unicorn solution I am not aware of?

1 | 2 | 3

20141 posts

Uber Geek

#1732502 7-Mar-2017 13:45

In terms of an actual backup, I just clone a drive once a week. So if I do delete a file, I can find it on the backup.

shk292

2853 posts

Uber Geek

Lifetime subscriber

#1732529 7-Mar-2017 14:12

Is it worth examining what the TB of data consists of and whether it needs to be backed up? Eg if it's all downloaded movies (or "Linux ISOs" to use the polite phrase), or rips of CDs/DVDs/BDs, then it doesn't need to be backed up.

You could separate it into replaceable and non-replaceable, and only back up the latter

jmorgannz

23 posts

Geek

#1732532 7-Mar-2017 14:16

@shk292 -

Yep a valid point. A lot of it may not need to be backed up; but a lot of it is things like machine images, which are not so easily segregated.

But there's also the time that it is going to take me to sort through it all; which I plan to do, but during which time I am in a danger zone.
In fact what prompted this is that I am going to be a lot more active with the data - organising and tidying / deleting - during which period it is much more likely I could instigate a situation where I need a backup!

@mattwnz -

Disks don't really seem like a viable long term backup medium to me.

I have had experiences where archived/stored disks fail to spin up after just sitting for a decent period.

Dynamic

3866 posts

Uber Geek

ID Verified

Trusted

Lifetime subscriber

#1732536 7-Mar-2017 14:29

I would encourage you having a quick trawl through your data and separating it into two or three tiers of importance. The most important would be backed up daily by CrashPlan or similar, the mid-tier stuff being backed up by an occasional copy to a large external HDD (or two) that are kept off site, and the least valuable stuff being protected only by the mirror. The least valuable stuff is probably just there because you have the space, but you would in reality never access it.

If (for example) you are a photographer and have lots of client images, charge them for the storage or give them their images on a 50Gb blu-ray disc (and keep a second copy yourself?) so long term storage is not your problem.

“Don't believe anything you read on the net. Except this. Well, including this, I suppose.” Douglas Adams

Referral links to services I use, really like, and may be rewarded if you sign up:
PocketSmith for budgeting and personal finance management. A great Kiwi company.

MadEngineer

4271 posts

Uber Geek

Trusted

#1732538 7-Mar-2017 14:31

mattwnz:
In terms of an actual backup, I just clone a drive once a week. So if I do delete a file, I can find it on the backup.

that's all fun and games till you realise you've backed up a bunch of corrupted or changed data

Imagine accidentally deleting half the pages of a book you've been writing and not knowing half the document is lost till a few weeks later.

You're not on Atlantis anymore, Duncan Idaho.

Talkiet

4792 posts

Uber Geek

Trusted

#1732541 7-Mar-2017 14:38

I have lots of stuff to backups, over a decade of sports photography all in RAW format (and jpgs, and other files) in total there is about 4TB of data I want backed up REALLY well.. .There's another 5-6TB I backup too, but not as strictly.

I have a cheap 4TB WD NAS drive with an external 5TB drive attached to that, giving me 9TB of network attached backup storage on a UPS. Everything is synced there daily.

I also have a fireproof safe with a waterproof clip-lock container where I store my local HDD backups that I refresh every month or 2.

Finally, I use backblaze.com which maintains a live backup of all my photos and everything else except movies and TV etc...

It's also worth mentioning that I rotate my spinning disks every so often - it used to be every 18 months but I am stretching that to just over 2 years at the moment. The larger good drives from my main PC go into the Safe backup pool, the local NAS drives and local PC drives get sold/given away and I put new drives in my PC and NAS.

There isn't an economical consumer level backup for lots of data (multiple TB) except for external HDDs. Get yourself an esata or USB3 cradle and a few big bare sata disks. If you're not going to run an online backup as well, maintain two offline backups and store one at home and one at another physical location.

Oh, I got my first computer in 1984 and I have never lost any data. I also use to work building (and testing) storage and backup systems for Sun machines for govt departments, universities and ISPs in the late 90's/early 2000s.

Cheers - N

Please note all comments are from my own brain and don't necessarily represent the position or opinions of my employer, previous employers, colleagues, friends or pets.

jmorgannz

23 posts

Geek

#1732543 7-Mar-2017 14:40

Hmm.

I wonder, renting a tape-drive seems like it could be an option for long time archival of the entire data-set.
Of course, recovery would require re-hiring, but that's probably acceptable.

Equinox IT

Cloud spending continues to surge globally, but most organisations haven’t made the changes necessary to maximise the value and cost-efficiency benefits of their cloud investments. Download the whitepaper From Overspend to Advantage now.

MadEngineer

4271 posts

Uber Geek

Trusted

#1732592 7-Mar-2017 15:30

Upgrade your data plan for a month if you want to go the online route.

Has anyone used the cross-site mirror option provided by readynas units? Would be awesome if you had a family member also wanting such a setup, then theoretically you could mirror each other's backups offsite.

You're not on Atlantis anymore, Duncan Idaho.

davidcole

6029 posts

Uber Geek

Trusted

#1732598 7-Mar-2017 15:40

Like others, split what it most important out, start with what's most important and give that to some online service.

I use crashplan, backing up from one server (any other machines copy their data to the server and that backs it up). At the time I seeded my primary backup I was on ADSL (1mbps upload yo!!). And it was a case of every couple of days or so, I'd add something else to the backup and let that tick over - while managing my usage (there was no unlimited then).

If you can bite the bullet and go unlimited for a month or so, and upload as much as possible.

I also had a secondary machine, and a bunch of in machine backups as cover while the cloud backup was running.

Previously known as psycik

Home Assistant: Gigabyte AMD A8 Brix, Home Assistant with Aeotech ZWave Controller, Raspberry PI, Wemos D1 Mini, Zwave, Shelly Humidity and Temperature sensors
Media:Chromecast v2, ATV4 4k, ATV4, HDHomeRun Dual
Server Host Plex Server 3x3TB, 4x4TB using MergerFS, Samsung 850 evo 512 GB SSD, Proxmox Server with 1xW10, 2xUbuntu 22.04 LTS, Backblaze Backups, usenetprime.com fastmail.com Sharesies Trakt.TV Sharesight

Lias

5589 posts

Uber Geek

ID Verified

Trusted

Lifetime subscriber

#1732614 7-Mar-2017 16:04

I use Crashplan

I'm a geek, a gamer, a dad, a Quic user, and an IT Professional. I have a full rack home lab, size 15 feet, an epic beard and Asperger's. I'm a bit of a Cypherpunk, who believes information wants to be free and the Net interprets censorship as damage and routes around it. If you use my Quic signup you can also use the code R570394EKGIZ8 for free setup.

cadman

1014 posts

Uber Geek
Inactive user

#1732629 7-Mar-2017 16:35

Lias:
I use Crashplan

Me too. 10TB on there. It's great although there are some pitfalls.

If you go with the free Crashplan you can just use a local drive to backup or even use a friend's Crashplan code to back up to a drive you give them to set up on their computer. Do the initial backup by going aroundown there with your computer then it's just small amounts of changed data after that.

richms

28168 posts

Uber Geek

Trusted

Lifetime subscriber

#1732660 7-Mar-2017 17:42

Crashplan seem to have built it to stop you using it for large data sets - will just sit synchronizing block information for days which the PC has to be left on for, then push a little bit up to the cloud, then start pruning things, then repeat the process only getting a few 100 gigs up a week at best.

Great on small sets where that carryon doesnt happen.

Richard rich.ms

cadman

1014 posts

Uber Geek
Inactive user

#1732675 7-Mar-2017 18:11

richms:

Crashplan seem to have built it to stop you using it for large data sets - will just sit synchronizing block information for days which the PC has to be left on for, then push a little bit up to the cloud, then start pruning things, then repeat the process only getting a few 100 gigs up a week at best.

Great on small sets where that carryon doesnt happen.

I didn't have that issue. I did my initial backup in under 6 months on a 10Mbps VDSL connection, only because the international speed (Australia) was peaking at about 6Mbps. It did a bit of "Synchronising..." but it wasn't overly intrusive to the backup process.

nigelramsay

80 posts

Master Geek

#1732737 7-Mar-2017 21:00

I used to use Crashplan, but now use Arq. It's a standalone backup utility that can connect to a number of online backup locations - AWS, Google Drive, Dropbox, etc. Also does any local folder or NAS, as well as generic SFTP.

It's good to not be dependent on any online service that can adjust their prices as they want to, or possibly go bust. I used to use PictureLife for all by photo backups, but then they went bust and it was VERY HARD to get my photos off them. As everyone else also wanted to do the same. That doesn't scale.

So yeah - Arq is awesome.

timmmay

20575 posts

Uber Geek

Trusted

Lifetime subscriber

#1732816 8-Mar-2017 06:59

Incremental backups are essential IMHO. They protect you against corruption, viruses, etc. I use a couple of different programs and methods:

Cobian backup for my large images and videos. I have many TB of this, and I don't really want them smooshed into a few huge files. Cobian keeps the files intact, and if it changes it just makes another copy. It uses more disk, but I'm fine with that because files are rarely changed.
AOMEI backup. It's a fairly standard backup program, incremental and differential, everything stored in large files. I put all my smaller information in here.
Macrium reflect to back up my OS disk (which is why I prefer smaller SSDs or system partitions)
My backups are stored offsite. I also have a second offsite location that I update rarely, and another backup in the detached shed.
CrashPlan for online backups of what I consider most critical. This was mostly documents, not my commercial images (too large), but since we had a baby I also back up personal images into Crashplan.

1 | 2 | 3