Geekzone: technology news, blogs, forums
Guest
Welcome Guest.
You haven't logged in yet. If you don't have an account you can register now.


timmmay

16518 posts

Uber Geek

Trusted
Subscriber

#238017 29-Jun-2018 05:40
Send private message

I run an Amazon Linux server as my production web server, running Wordpress and a photo gallery. AWS provides a good way to restore the server if it fails (I take automatic weekly EBS snapshots), but I want an offsite backup of my content and configuration in case my AWS account becomes inaccessible, is lost due to error, etc. Not much changes on my server day to day - Wordpress updates, a bit of new content, but the average delta is in the low MB.

 

I currently use Borg Backup. Borg creates deduplicated block level backups locally, which I upload to S3 using the AWS command line. The problem with Borg is every time it runs it creates a new file with all of the data, not just the new data. It does this in an efficient way, appending information, but it means I effectively upload all my data to Amazon S3 every night, which increases my bandwidth charges.

 

Here are my requirements:

 

  • Suitable for nightly backup on Amazon Linux (no GUI, console only)
  • Memory efficient - I run a t2.nano, which is 0.5GB of RAM with 0.5GB swap. Right now it has 100MB RAM free (once disk cache is emptied) and 470MB swap free. I can tweak PHP a little more to make another 100MB of RAM available.
  • Bandwidth efficient - should create incremental backup files each day, without renaming the existing files.
  • Disk efficient - ideally doesn't keep a full second copy of all data on the disk, but I do have enough disk space for this if required.
  • Incremental backups, with a schedule to remove old backups.
  • Includes Backup scheduling or is compatible with cron.
  • Must not rely on a backup server to control or receive backups, or backing up to a computer over ssh. It must be able to back up to a simple file store like Amazon S3. This counts out quite a few products.
  • Not too difficult to install (I can build from source but prefer to get a package / release).

 

 

Here's what I've considered

 

Duplicati

 

Duplicati seems like great software with a lot of potential, though there is at least one known restore bug when you use non-standard block sizes - which I don't need to do with my web server. The developer says it's an interface bug, there's no problem with the core of the system, but it's been outstanding for more than a year. I think it fulfills all my requirements above, though I'd have to check about memory efficiency. It has a nice web interface.

 

It relies on extra packages like Mono, which I can get installed on Amazon Linux, but it's not totally trivial.

 

This is probably my top pick, if I can get it working without too much effort.

 

Duplicity

 

Duplicity looks interesting. It's stable and "fairly mature" and is released fairly regularly. It basically uses rsync and tar to create delta files and can upload them to S3. It's on my list to try as it's meant to be fairly stable.

 

ZBackup

 

I thought ZBackup looked quite promising, but there hasn't been any development for about three years. That might indicate it's stable, but the text on the website suggests some work is required and no-one has bothered. This is probably my third choice.

 

Cloudberry Backup

 

I use CloudBerry backup for my PC, and they offer a Linux edition. I don't totally trust CloudBerry, for reasons I can't quite put my finger on, so I keep an incremental backup and a mirror when I use their software. When I use CB to back up to S3 I use their "sync" feature, and rely on S3 for versioning and encryption.

 

I don't like that instead of large backup files they tend to create one backup file for each file being backed up, plus delta files, which isn't disk space efficient. They do have an option that lumps everything together, but it has limitations. Their deduplication algorithm is pretty poor in my testing.

 

However, having said that, my restore tests have always been successful, it's fairly mature, and it's a commercial supported product which means it's likely tested reasonably well. But because of my gut feel I'd prefer something else.

 

 


Create new topic
nzkc
900 posts

Ultimate Geek


  #2045835 29-Jun-2018 07:38
Send private message

You might want to have a look at https://rclone.org .

Im using this to sync photos on a Linux machine into Bacblaze B2 storage. It was very simple to set up and use. Has both rpm and deb packages.

timmmay

16518 posts

Uber Geek

Trusted
Subscriber

  #2045862 29-Jun-2018 08:42
Send private message

Thanks @nzkc. That looks more like a sync program, whereas I want incremental backups and versioning - which I just added to my list of requirements.

 

One part of my home backups sync's to Amazon S3 and uses S3 versioning, which is ok. I do like the feature of Borg that tells it to keep daily backups for a week, weekly backups for 3 months, then monthly backups forever (or something like that). That's not essential though.


 
 
 
 


marpada
337 posts

Ultimate Geek


  #2046156 29-Jun-2018 18:42
Send private message

timmmay:

 

Duplicity

 

Duplicity looks interesting. It's stable and "fairly mature" and is released fairly regularly. It basically uses rsync and tar to create delta files and can upload them to S3. It's on my list to try as it's meant to be fairly stable.

 

 

Duplicity is ok but might be painful to run on a server with low memory. If duplicity is killed by the oom_killer it leaves a lock so the next scheduled backups will fail to run until you delete the lock. Might be ok if you don't have lots of files to backup though.

 

I looked for a reliable solution that could backup gazlillions of small files to S3/Backblaze for a while and ended up using borg + rsync.net (an ssh-like store provider with some competitive pricing but unfortunately not pay-as-you-go).

 

 

 

 

 

 

 

 


timmmay

16518 posts

Uber Geek

Trusted
Subscriber

  #2046197 29-Jun-2018 19:42
Send private message

marpada:

 

Duplicity is ok but might be painful to run on a server with low memory. If duplicity is killed by the oom_killer it leaves a lock so the next scheduled backups will fail to run until you delete the lock. Might be ok if you don't have lots of files to backup though.

 

I looked for a reliable solution that could backup gazlillions of small files to S3/Backblaze for a while and ended up using borg + rsync.net (an ssh-like store provider with some competitive pricing but unfortunately not pay-as-you-go).

 

 

Thanks for the thoughts. I already use Borg + S3 and it has that limitation which is annoying and really quite bizarre as it will take additional network / internet bandwidth for everyone.


dr0pknutz
5 posts

Wannabe Geek


  #2047936 3-Jul-2018 00:02
Send private message

Take a look at restic - https://restic.net

mdf

mdf
2683 posts

Uber Geek

Trusted
Subscriber

  #2048077 3-Jul-2018 10:06
Send private message

Like @nzkc I use rclone to B2. I am far from an expert in this (far!) but you might like to dig around in the commands/forums for rclone since I _think_ it might do want you want, albeit by having to fiddle in the CLI with the various flags and commands.

 

For me, I prefer rclone's sync/copy/move approach (keeping the original file names and structures) rather than duplicity's approach of making a single huge monolithic file that you (essentially) have to use duplicity to restore. rclone will do the monolithic encrypted file too though using the crypt command.

 

My current approach is pretty simple (I'm using Backblaze's built in versioning and don't-really-delete-just-hide approach), but while I was experimenting I had scripts scheduled to run daily, that would only check things that had changed in the last day. Nothing deleted from my live folder would be deleted on the backup server, instead moved to a backup directory with appropriate date and time stamps. Backup directories older than a month were then deleted.

 

rclone doesn't support binary diffs (apparently it might actually be able to do it, but the remote storage providers don't). Incremental backups therefore definitely don't work for monolithic files. However, I've found incremental backups of _directories_ with many smaller files to work really well (though I appreciate we might have a different understanding of what incremental backups actually means...).

 

Nick at rclone is also one of the friendliest and most helpful developers I have had the pleasure of dealing with. If it comes close to what you want, he's very response to feature requests. 


timmmay

16518 posts

Uber Geek

Trusted
Subscriber

  #2048115 3-Jul-2018 10:23
Send private message

@dr0pknutz Restic looks good from the home page. I'll have to have a read at some point when I have time.

 

Duplicati fixed the year old restore bug. I got Duplicati working on Amazon Linux, did a restore test - it works on Linux but fails on Windows. So I found another restore bug. It's probably a web interface issue but I've had nothing but problems every time I've played with Duplicati.

 

I prefer incremental backups rather than versioned files, as it can save significant bandwidth and storage space. That's usually implemented as large files.


 
 
 
 


stinger
628 posts

Ultimate Geek
Inactive user


  #2048177 3-Jul-2018 11:26
Send private message

I use duplicity with BackBlaze (but they also have AWS support as well). I do a full backup every month, and incremental backups every 2 hours. I keep two full backups so I have between 30-60 days of rollback, which has saved my bacon a couple of times. Duplicity encrpyts (GPG) your files, so no matter how hard Backblaze try, they cannot see anything, which was a key thing for an online backup solution.


timmmay

16518 posts

Uber Geek

Trusted
Subscriber

  #2048543 3-Jul-2018 20:22
Send private message

Restic seems pretty good. Really simple to use, good prune policies, works with S3. I've started looking at that and will play some more as I have time. It works on Windows too, so I might give that a shot.

 

I'm less than thrilled with CloudBerry on Windows, which has stopped working, and I need to rebuild the database or do some SQL they've given me to get it working. Duplicati I'm also not thrilled with in general.


Create new topic





News »

Nanoleaf enhances lighting line with launch of Triangles and Mini Triangles
Posted 17-Oct-2020 20:18


Synology unveils DS16211+
Posted 17-Oct-2020 20:12


Ingram Micro introduces FootfallCam to New Zealand channel
Posted 17-Oct-2020 20:06


Dropbox adopts Virtual First working policy
Posted 17-Oct-2020 19:47


OPPO announces Reno4 Series 5G line-up in NZ
Posted 16-Oct-2020 08:52


Microsoft Highway to a Hundred expands to Asia Pacific
Posted 14-Oct-2020 09:34


Spark turns on 5G in Auckland
Posted 14-Oct-2020 09:29


AMD Launches AMD Ryzen 5000 Series Desktop Processors
Posted 9-Oct-2020 10:13


Teletrac Navman launches integrated multi-camera solution for transport and logistics industry
Posted 8-Oct-2020 10:57


Farmside hits 10,000 RBI customers
Posted 7-Oct-2020 15:32


NordVPN starts deploying colocated servers
Posted 7-Oct-2020 09:00


Google introduces Nest Wifi routers in New Zealand
Posted 7-Oct-2020 05:00


Orcon to bundle Google Nest Wifi router with new accounts
Posted 7-Oct-2020 05:00


Epay and Centrapay partner to create digital gift cards
Posted 2-Oct-2020 17:34


Inseego launches 5G MiFi M2000 mobile hotspot
Posted 2-Oct-2020 14:53









Geekzone Live »

Try automatic live updates from Geekzone directly in your browser, without refreshing the page, with Geekzone Live now.


Support Geekzone »

Our community of supporters help make Geekzone possible. Click the button below to join them.

Support Geezone on PressPatron



Are you subscribed to our RSS feed? You can download the latest headlines and summaries from our stories directly to your computer or smartphone by using a feed reader.

Alternatively, you can receive a daily email with Geekzone updates.