Musings on rolling your own remote "cloud" backups and general backups

Forums › IT Pro and developers › Musings on rolling your own remote "cloud" backups and general backups

1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9

Amanzi
1355 posts

Uber Geek
+1 received by user: 335

ID Verified

Trusted

Lifetime subscriber

#1787073 23-May-2017 21:08

I was impressed with Arq. Yes, the UI needs work but I like that he publishes the backup format specification and even an open source application to restore information from backups. So even if he gives up the software, you'd never be stuck. What I didn't like is the cost - I quickly realised that I'd need at least a TB of S3 storage which costs more than a monthly Crashplan Family account. So for me, I'm sticking with Crashplan Family account - also means I can backup my in-laws' computer as well as the four other home computers I have on there, and still have room for 5 more computers if needed.

timmmay

20859 posts

Uber Geek
+1 received by user: 5350

Trusted

Lifetime subscriber

#1787080 23-May-2017 21:17

amanzi:

I was impressed with Arq. Yes, the UI needs work but I like that he publishes the backup format specification and even an open source application to restore information from backups. So even if he gives up the software, you'd never be stuck. What I didn't like is the cost - I quickly realised that I'd need at least a TB of S3 storage which costs more than a monthly Crashplan Family account. So for me, I'm sticking with Crashplan Family account - also means I can backup my in-laws' computer as well as the four other home computers I have on there, and still have room for 5 more computers if needed.

On S3 you can use a lifecycle rule to move it to Glacier storage class, reducing the cost by 1/3. You'd have to check if Arq works with that, but I think it does. 1TB is $4/month.

Yoban

453 posts

Ultimate Geek
+1 received by user: 86

#1787385 24-May-2017 12:27

Hi all, valuable discussion.

Have you considered http://www.todo-backup.com/products/home/free-backup-software.htm from EaseUS. I have been using it for some time on 2 laptops and a desktop with backups being written to local disk and NAS and then the NAS is sync'd with amazon cloud drive.

have not had any issues and has all the features I need so far.

timmmay

20859 posts

Uber Geek
+1 received by user: 5350

Trusted

Lifetime subscriber

#1787405 24-May-2017 12:52

Yoban:

Hi all, valuable discussion.

Have you considered http://www.todo-backup.com/products/home/free-backup-software.htm from EaseUS. I have been using it for some time on 2 laptops and a desktop with backups being written to local disk and NAS and then the NAS is sync'd with amazon cloud drive.

have not had any issues and has all the features I need so far.

I've looked at a few traditional backup systems. They don't back up directly to cloud, so I'd need another piece of software for that. I'd like to get rid of CrashPlan because I see it as an attack vector waiting to be exploited, whereas AWS I can easily control and lock down.

CloudBerry Backup actually does 5TB to S3, 5TB to Glacier, and unlimited backup to local disks. Because of that I'm going to evaluate that next. It's very flexible and should do everything I want... hopefully.

MikeAqua

8031 posts

Uber Geek
+1 received by user: 3820

#1787450 24-May-2017 13:41

timmmay:

Mike, that will give you incremental, versioned backups right? If a virus on your PC encrypts everything on the NAS then backs it up before you notice, will you be able to roll back versions?

My reading so far suggests yes it will - with a bit of initial setup.

Mike

ripdog

548 posts

Ultimate Geek
+1 received by user: 373
Inactive user

#1787519 24-May-2017 15:50

I'm not sure you should think of backup as 'setup and forget'. Even the best backup schemes tend to fail occasionally, so they are something you really should be doing test restores from at least once a year (world backup day is a good time). IMO this negates your issues of Arq being a one-man-band, since this is something you can check when you check your backups, and if Arq is dead, you can take that opportunity to set up something else.

Lenovo

Shop now for Lenovo laptops and other devices (affiliate link).

timmmay

20859 posts

Uber Geek
+1 received by user: 5350

Trusted

Lifetime subscriber

#1787805 24-May-2017 21:00

I've had a bit of a play with CloudBerry Backup this evening. So far I'm quite impressed.

It backs up to file systems just fine. Take the disk out, put it in, no problem.

It backs up to S3 fine.

There are three major backup modes:

Advanced, which basically copies each file, then if it's changed saves a diff. It's meant to be block level, but I'm not convinced it is (see below). An advantage of this when backing up to S3 is the upload is fast - 8 to 12Mbps to the USA on my 20Mbps upload connection.
Archive, which writes data to a single large file. Problem with this one is a full backup (which you should do occasionally) really does a full backup, and doesn't prune old files. So it takes a lot more space. The advantage of it is on cloud storage providers you're often charged for each PUT operation (one per file at least), so it can be cheaper to have one large file. The one file is uploaded in a single thread so it's slow, usually 2Mbps to the USA on my 20Mbps connection.
Simple, which copies data but doesn't compress or do versions

It doesn't seem to be very smart at backing up changed files. I backed up a 70MB file, changed a few bytes, backed up, changed a few more bytes, and backed up again. The diff files were 3.5MB and 32MB. I did the same thing again, with both archive and advanced mode backups but changing bytes near the start of the file, and it just backed the whole thing up again. I've asked support why. I don't change files all that often, so it's probably not a big deal.

It has plenty of features around backup versions, deleting versions, validation of backups, encryption, etc. More than most. It backs up to a staggering number of services - S3, Glacier, sftp, local disks, Dropbox, Google Drive, One Drive, Oracle Cloud, sFtp, BackBlaze, Azure, Google Cloud, and many more.

The user interface is good, at least compared with others. Working out how to delete backup destinations and validate backups took some time.

I'm not 100% impressed with the technical engine behind it, with regards to deduplication and such. I think it's good enough, but could be improved on a few fronts to improve efficiency. Duplicati is significantly better at this, but has faaaar too many bugs to be a viable backup tool.

So right now, CloudBerry backup is looking like it will be my backup tool. My annual CrashPlan subscription expires in less than a week. I think I might switch to a monthly plan for a few months, do plenty of testing and test restores, then turn CrashPlan off. I trust AWS storage more, because I can absolutely prevent data being deleted. With CrashPlan a virus could send a simple command to delete files or the whole account, and I have no idea if CrashPlan would be able to restore. I might ask them.

In CrashPlan, when you deselect files from your backup it says "When you remove files and folders from your backup selection you will permanently delete the files from your backup archive, and you cannot restore, including all previous versions.". That suggests a virus that targets CrashPlan customers could lose all their data.

timmmay

20859 posts

Uber Geek
+1 received by user: 5350

Trusted

Lifetime subscriber

#1787879 25-May-2017 08:42

A small update.

Arq have released a new version to address the bug I found. It took them around 36 hours. I'll test it tonight.

CloudBerry answered my support request, but it looks like a low level support person has dashed off some basic answers without really thinking them through fully. They also haven't attempted to reproduce what could be bugs or areas that could need enhancement. I've followed up with them.

I've contacted CrashPlan to suggest an enhancement that would protect against a virus that specifically targets CrashPlan. The support person thinks it's unlikely to happen, whereas I think it's only a matter of time. I've gone back suggesting they send the information to someone higher level, and I've suggested a few possible mitigations.

I'm going to go forward with CloudBerry for now. I need to understand their backup modes better, and do some testing on restores.

ripdog

548 posts

Ultimate Geek
+1 received by user: 373
Inactive user

#1790580 28-May-2017 11:35

Heh. One app's developer fixes a bug you reported and pushes a new version to you in *36 hours* and you go with the app backed by a support department which ignores you and reads off a script to you. I don't get it, mate. What are you going to do when you find a major bug in cloudberry and the support department ignores you again? If you don't think they'd ignore a bug you're reporting, have a look at Microsoft's labyrinthine support procedures when this guy tried to report a bug in excel: https://news.ycombinator.com/item?id=14426511

It's the same attitude.

Oh, and Arq does block-level backup perfectly. I know, I have a 700GB file.

timmmay

20859 posts

Uber Geek
+1 received by user: 5350

Trusted

Lifetime subscriber

#1790586 28-May-2017 11:46

Actually I've steered away from CloudBerry and I'm doing a detailed evaluation of Arq. I've reported one major show stopper bug, it was fixed in 36 hours or so. I've reported a few minor bug, and around a dozen suggestions. It seems technically good but has many, many areas that could be improved on - some usability, some performance.

I do think that a one man band for software does introduce risks to backups. What if the next windows10 update breaks the backup and restore code right after he gets hit by a car? You'd have to roll back to an older OS. Sure, it's unlikely. I don't trust my backups to any one format or media though, so for example I have files in Glacier, backups in Arq, and plain copies on disks.

Arq validating backups is exceptionally slow, it only uses one core. On a 400GB backup I gave up the validation after a few hours, but I think one reason it was going slow was because it'd decided to constantly write massive debug logs - even though the log level was set to info.

Arq usability could use some work. I have an external disk with four backup sets defined, as I figure it reduces risk to have them separate. I have to hit "backup now" on the first, wait for it to finish, then repeat for each backup. So I can't backup overnight, I have to be there. I could schedule it I guess, but then I'd get errors.

I also found that if Arq was running a manual backup at the time a backup is scheduled it doesn't queue it up, it just ignores it. Though I haven't tested it, it's observation of one time, and it could be that there was nothing new to back up in that set.

The Arq documentation is somewhere between poor and non-existent. It doesn't look like it's been updated in years. Compare that to CloudBerry which has extensive documentation and a huge number of blog posts going back years. CloudBerry inspired much more confidence, but has a few flaws that led me towards Arq. I'm reconsidering it though.

All in all Arq is looking pretty good, but I'm surprised it got to Version 5 with such a primitive interface.

timmmay

20859 posts

Uber Geek
+1 received by user: 5350

Trusted

Lifetime subscriber

#1793746 2-Jun-2017 14:36

Intro

I've come to what I think is a conclusion and my final position for backups. I'll evaluate for a few months then probably get rid of CrashPlan - which I switched from annual to monthly payment as my annual subscription was up.

I've spent a LOT of time thinking about backups and experimenting with back software and online backup providers - probably 40 hours. I've written a fairly thorough review of some backup products for my blog. If anyone wants to read it PM me, I don't tend link from GZ to my websites.

Backup Requirements

My general requirements for backup software are:

I want primary one tool that is capable of meeting almost all of my backup needs. Simplicity is a good thing. I'll actually use a different tool for Glacier, to spread my risk.
Software should backup to a range of destinations. That includes cloud backups, connected disk backups, and backups to external disks.
The features must include incremental backups or block based backups with a version history, encryption, compression, scheduled backups. Ideally it should include de-duplication and a range of options to purge old versions.
Client side encryption is important to have.
Should have sufficient options for a range of backup jobs. This could include include/exclude masks, the option not to back up subfolders, flexibility on when to purge old versions, etc.
The tool should be well documented. Ideally there should be blog posts and / or a user community to help with issues.
The tool should be reasonably quick to run backups. I don’t sit there watching them run, and many backups are scheduled, but faster is generally better than slower.
If something goes wrong I want there to be help available, either paid or community.
The tool should be reasonably easy to use, with a sensible interface. For example backups should be able to be either queued or run in parallel.

Why I chose CloudBerry

I've decided CloudBerry backup the best tool for me to meet those requirements. The key reasons are:

CB backs up to pretty much any destination, quickly and efficiently. It can run multiple backups in parallel, scheduled or manual.
CB is backed by a company, not a person. Arq is a one man band.
CB provides support and has MASSES of documentation. Arq is basically undocumented.
CB has a simple, clear user interface that's easy to use.
CB is significantly faster than Arq to backup. It's miles faster than CrashPlan when you're talking about external disks, because it doesn't validate them every time you connect a disk.
CB deduplication is really quite poor compared with Arq or Duplicati, but I don't tend to have duplicate data, and large files very rarely change. Overall CloudBerry is "ok" technically, I think it's reliable but not completely optimised yet.

Why I also chose Glacier

CloudBerry backs data up to S3, which is more expensive than Glacier. I have a lot of static data, like old photos and family videos, so they can go into a cheaper archiving solution rather than more expensive "hot" storage like S3.

Why I ruled others out

I ruled Duplicati out because restores failed, but I think in 1-2 years it will be awesome. I ruled CrashPlan out because it's very slow with external disks and is theoretically vulnerable to being completed wiped by a virus. I may actually do a proof of concept on that if I ever have free time. Arq was almost my choice, but the poor UI, one man band, and really poor documentation ruled it out.

Cost

So far it's going fine. Backups to internal and external disks work fine. Backups to S3 and BackBlaze B2 are both fast. B2 gives you 10GB free and seems nice, much easier to use than S3 for a beginner. My home based external disk backups are fine. I'm about to delete the Duplicati backups from my main external backup and replace them with CrashPlan this weekend.

I'm storing 60GB of data in AWS Glacier. That's all my documents, 2000 pixel wide copies of all my photos, and 720p versions of all my videos, everything, with PAR2 files for some redundancy. Glacier costs me US$0.29 per month, around NZ$5 per year. I used free software to upload it, FastGlacier and / or CloudBerry Explorer. I have the Glacier backup set up using a different AWS IAM user from S3 so if credentials are leaked I can't easily lose S3 and Glacier backups.

I have all my documents, jpeg versions of all my photos, and the last couple of months RAW files in S3. Once my data is backed up to external disk and glacier I'll remove the raw files from S3 to reduce cost. I have 9.5GB of data in S3 Infrequently Access class storage, which is costing US$0.12 per month, around NZ$2 per year I guess. I could significantly increase what I store there for very little money.

I think it'll end up costing me less than NZ$10 per year for my backup storage. It'll take more of my time than CrashPlan did though, to maintain and check.

Security, Encryption and Versions

When I set up CloudBerry backups I have everything encrypted on the PC before it's sent to Amazon S3. This is more secure, but means any bugs in CloudBerry or if the software stops working I may lose access to my data. It also means you have to install CloudBerry Backup to restore files.

S3 provides server side encryption with AWS managed keys, which is secure enough for fortune 500 companies and government. S3 also provides versioning. CloudBerry will let you simply sync your data to S3.

I may change my backups so that nothing is encrypted on the client, but it's encrypted on S3 by AWS. This reduces my risk of CloudBerry problems, and only slightly increases my risk given S3 is enterprise grade.

Disclaimer

Disclaimer: I discovered after choosing CloudBerry Backup that I could get a free copy of a CloudBerry product because I wrote a review on my blog. I'd have paid for it myself if I didn't get a free copy. I also got a free copy of CloudBerry Explorer for S3 / Glacier as they give a license to anyone AWS Pro certified.

Quic Broadband

Move to New Zealand's best fibre broadband service (affiliate link). Free setup code: R587125ERQ6VE. Note that to use Quic Broadband you must be comfortable with configuring your own router.

rb99

3505 posts

Uber Geek
+1 received by user: 1830

Lifetime subscriber

#1796551 8-Jun-2017 11:55

Does anyone happen to know with Amazon Cloud Drive uploading using the desktop app, it says -

To upload files:

Select the file(s) or folder(s) from your computer that you want to upload.
Move the file(s) or folder(s) into the Amazon Drive folder on your computer.

Are you moving the actual file to the amazon drive folder on the computer or are you just putting the existing location of the file you want to back up into the desktop app so it knows where to get it from. Really don't want to move actual files around if possible.

“The modern conservative is engaged in one of man's oldest exercises in moral philosophy; that is, the search for a superior moral justification for selfishness.” -John Kenneth Galbraith

rb99

timmmay

20859 posts

Uber Geek
+1 received by user: 5350

Trusted

Lifetime subscriber

#1796552 8-Jun-2017 11:57

Most of the cloud sync tools (Dropbox, Onedrive) require you to copy your files into the folder that it monitors and uploads.

This is one advantage of using software like CloudBerry, Arq, etc. You can tell it what files you want uploaded, where to, how often, and set version retention and deletion policies.

rb99

3505 posts

Uber Geek
+1 received by user: 1830

Lifetime subscriber

#1796557 8-Jun-2017 12:14

Seeing as I'm probably too cheap to pay for Cloudberry, and I saw you quite liked ARQ, I guess that might be worth a try if it'll upload files to Amazon Cloud without actually having to move them - its a better substitute for Amazon Drive Desktop in this case ? My needs a quite simple, probably not to worried about versioning for instance.

“The modern conservative is engaged in one of man's oldest exercises in moral philosophy; that is, the search for a superior moral justification for selfishness.” -John Kenneth Galbraith

rb99

timmmay

20859 posts

Uber Geek
+1 received by user: 5350

Trusted

Lifetime subscriber

#1796583 8-Jun-2017 12:35

rb99:

Seeing as I'm probably too cheap to pay for Cloudberry, and I saw you quite liked ARQ, I guess that might be worth a try if it'll upload files to Amazon Cloud without actually having to move them - its a better substitute for Amazon Drive Desktop in this case ? My needs a quite simple, probably not to worried about versioning for instance.

Arq and CloudBerry are both paid products. Arq is US$50, CloudBerry desktop is $30. CloudBerry has a free tier that will backup 200GB no charge.

Either will let you backup from any part of your drive to your backup destination.

Alternately Duplicati is free, but not sure if it goes to Amazon Drive, and it's not a mirror it's a backup program. You can't directly access files on the cloud drive. Attic and Borg are both command line backup programs that have similar functionality but are more reliable than Duplicati.

1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9