Advice on home network setup using Proxmox VE and Freenas

Forums › Self-hosted › Advice on home network setup using Proxmox VE and Freenas

SumnerBoy

2068 posts

Uber Geek

ID Verified

Lifetime subscriber

#177126 23-Jul-2015 11:18

Hey GZers,

I currently have an Ubuntu 12.04 headless server running on an Intel i3-2120T with 4GB RAM. It runs a load of different apps and services including Freeswitch for VOIP, dnsmasq, and various home automation systems (mainly openHAB + mosquitto), plus MySQL and InfluxDb and a small web server.

I have a reasonable backup strategy in place, backing up all config files to a local external HDD and to the cloud using Crashplan.

However the complexity of the server is growing as I add more bits and pieces, and it is at the point now that if it crashed and I had to rebuild from scratch I can see it taking me weeks to get it back to its current state.

So I am looking into some better disaster recovery options and have stumbled across Proxmox VE (after a recommendation). So having done a load of reading I have come up with the following strategy, and just wanted to see if any experts out there had any advice or suggestions about what I am planning, and if there is anything I should be doing differently.

* buy a new box for hosting Proxmox VE
- i5-4590 CPU (quad core with VT)
- ASUS H97M-Plus MOBO
- 16GB RAM
- 2x1TB HDDs in RAID1 (toyed with the idea of SSDs but too expensive and not needed for this low-IO box) - this drive will just be used for the VM OSs and local storage.

* install Proxmox VE on this new box

* create a series of OpenVZ containers (not KWMs since I don't need multiple OS's - linux is fine)

* have a container for databases (MySql, InfluxDB, redis), media (Squeezeserver etc), apps (openHAB, mosquitto etc), monitoring (collectd and other scripts), web (apache2 web server w/ ownCloud etc).

Having built all these and tested the snapshot/backup/restore of the containers to make sure I am happy with the recovery process I will then decommission the old server and turn it into a stand alone NAS running Freenas.org.

It already has a 2TB WD Green HDD so I plan to buy two more 2TB WD Red HDDs and configure all three in a RAID5 array to give me 4TB of storage. This will be the main data store area and all my VMs will be configured to access these via NFS.

There is also a 160GB HDD in the old server which I will use for the NAS OS.

This is my first attempt at a proper server config with RAID and VMs so I am reading a lot at the moment and trying to take it all in before spending all this money on the new box.

Therefore I wanted to run my plan by you guys and see if I am doing anything stupid, or if the hardware I have selected will give me any problems.

Also want to make sure it will be easy enough to do live snapshots/backups of the various VMs and file away using Proxmox so that if there is a disaster situation I can rebuild the server relatively quickly, without having to do complete re-install/re-config on all the apps and services.

Sorry for the long post - wanted to get as much detail as I could - but please fire away with any questions or suggestions.

Many thanks!
Ben

solival

160 posts

Master Geek

#1350453 23-Jul-2015 15:05

I recommend to avoid using OpenVZ. It is quite unstable. In addition, upcoming Proxmox 4 will change OpenVZ to LXC containers (thumbs up for that). If you need only containers, then it's better to just use LXC or Docker over Ubuntu Server 14.04.

You have quite bad backup strategy if in case of crash you will need weeks to recover. I use LVM snapshots for root filesystem that allows me to make simple tars backups (or just revert changes if things went wrong), /boot is on ext2, and full data backup using rsync (weekly). Last crash required about half a hour to fix. I just prepared LVM layout on new HDD, copied root and /boot filesystem from latest tars, chroot, grub-update and done.

About system. If you do RAID5, you probably will also need ECC. Especially if it works 24x7. If you need ECC, you need reconsider your CPU. It should be either i3 or entry level Xeon. AFAIK neither i5 nor i7 supports ECC. Check also motherboard for that.

In any case RAID does not cancel backups.

If you need virtualization, did you do research before you choose Proxmox? There are other solutions as well, for example ovirt (Red Hart) or vSphere Free (VMWare). They all have advantages and disadvantages so it's better to have a closer look on every of them.

SumnerBoy

2068 posts

Uber Geek

ID Verified

Lifetime subscriber

#1350497 23-Jul-2015 15:15

Thanks for those thoughts Solival. I have been looking into the various options but there is a LOT of information to digest, and being new to this side of networking/server config it is often a little overwhelming. Hence my post on here to get some expert advice.

I looked into Docker originally, but the networking side of things put me off. It looked quite complicated setting up containers to *speak* to one another. Whereas with Proxmox/OpenVZ containers, you just assign each one an IP address and away you go.

I am only planning to do RAID5 on my NAS which is the i3-2120T. The VM host will be the i5 which is using RAID1.

And yes, well aware the RAID does not remove the need for backups. But it does give piece of mind that if a drive fails the system will continue working. I plan to backup my VM snapshots to the NAS, and run backups from the NAS to external HDDs and the cloud via Crashplan.

The reason my current strategy is so poor is that if my OS drive failed I would have to reinstall/reconfigure everything from scratch. I have all /etc backed up so it wouldn't be *from scratch* I guess, but it would require a lot of work to get everything up and running again. That is why I want to go down the virtual route.

I hadn't come across those other solutions however, so I will investigate ovirt and vsphere, thanks for that.

SumnerBoy

2068 posts

Uber Geek

ID Verified

Lifetime subscriber

#1351226 24-Jul-2015 16:26

Thought I might get a bit more feedback on this - surely there are plenty of you out there running some pretty slick home server/NAS configs?!

Have changed my thinking a bit and might now go for 2x120GB SSDs in the new VM host, configured with software RAID1. 120GB should be plenty of local space for my VMs, anything large will be offloaded to the NAS via NFS shares. And they are only a few $$ more than the 1TB HDDs I originally spec'ed.

And I think I will run the FreeNAS from an 8GB USB stick as it seems the recommended way. This will free up the 160GB HDD which will be spare.

Still haven't quite figured out how the backup strategy will work but I am thinking something along the lines of backing up my containers to the NAS, and then using Crashplan running on the NAS to backup to attached external HDDs and up to the cloud. Just haven't quite got my head around how the container backup/snapshot/restore stuff all fits together.

Have done plenty of reading and it seems like Proxmox is pretty popular for both home use and in production environments. And it seems that OpenVZ is pretty stable as well - any reason you recommended avoiding it @solival?

chevrolux

4962 posts

Uber Geek
Inactive user

#1351249 24-Jul-2015 16:58

I have been running a vSphere box for a number of years now. Same sort of specs as what you are talking but with 24GB Ram and the OS is installed on to a SSD. There are then a bunch of spinning disks for the NAS VM.
I then run the following VM's on it:
- FreeNAS
- Ubuntu 14.04 - Web Server for Sick beard, Sabnzbd, uTorrent etc
- FreePBX Distro (Basically CentOS with Asterisk, FreePBX etc)

All VM's are run off the SSD and then for FreeNAS I just added the spinning disks to be used for the actual NAS storage.

After that I still have plenty of room to spin up other VM's as i want to, for mucking around with stuff before pushing to 'production' as it were. vSphere is such a simple system and there is ALOT of documentation so you can get as complex as you wish depending how much reading you are willing to do.

vSphere then creates Snapshots of the machines and they are saved to an external usb which I have two of and I swap them when I remember (normally once a week), one then gets hidden away and the next weeks worth of snapshots accumulate. Fairly rudimentary process but in the 4-5 years this box had been running I haven't had to restore from a snapshot due to a disk failure (user error is another story haha). But I do find that snapshots make things so easy to muck around with. before I screw with the 'production' VM's I just take a snapshot, spin up a new VM, play away and then reload the production machine from my dev snapshot once i am happy with it.

Oh and just a note, I have zero business experience with virtual machines (i do networks and phone systems) so have no idea if what I do at home is actually best practise.

SumnerBoy

2068 posts

Uber Geek

ID Verified

Lifetime subscriber

#1351251 24-Jul-2015 17:05

Thanks Chevrolux - so you actually run your NAS as a VM on the host, well the NAS OS at least, and then just point it to your HDDs for storage - didn't think of that! Not a bad idea for a home server - perhaps I don't need a separate box for the NAS... would simplify things a bit and reduce power consumption too. Plus it means the Freenas is easily backed up along with the other VMs.

Would just need to make sure my new VM host can house 3x2TB HDDs and 2x120GB SSDs tho...

Any recommendations for where to source the hardware? I have been dealing with the guys at PC Force and have found them pretty helpful, but open to other suggestions.

Mark

1653 posts

Uber Geek

#1351372 24-Jul-2015 20:59

Just to drop a suggestion, if you are going to go the FreeNAS with ZFS route you should be looking at ECC RAM in your system. Though if you are going to do regular backups you could save pennies and use non-ECC.

michaelmurfy

meow
13217 posts

Uber Geek

Moderator

ID Verified

Trusted

Lifetime subscriber

#1351375 24-Jul-2015 21:16

I'm actually using XenServer w/ several VM's running on an old HP Microserver. I am thinking of using Docker on it instead as a lightweight solution.

Docker is great once you get the hang of it and will do what you need just fine.

Michael Murphy | https://murfy.nz
Referral Links: Quic Broadband (use R122101E7CV7Q for free setup)

Are you happy with what you get from Geekzone? Please consider supporting us by subscribing.
Opinions are my own and not the views of my employer.

Wise

Send money globally for less with Wise - one free transfer up to NZ$900 (affiliate link).

SumnerBoy

2068 posts

Uber Geek

ID Verified

Lifetime subscriber

#1351376 24-Jul-2015 21:19

I did look at Docker Michael, but the networking side of things looked a bit fiddly. I like the idea of lightweight containers rather than full VMs, that is why I am looking at Proxmox with OpenVZ.

SumnerBoy

2068 posts

Uber Geek

ID Verified

Lifetime subscriber

#1351377 24-Jul-2015 21:22

Hey Mark - yeah it seems like the hardware requirements for Freenas are actually pretty heavy. They say min of 8GB of RAM! But there is plenty of anecdotal evidence of people happily running on 4GB.

I don't know enough about the ZFS - is it that much better?

michaelmurfy

meow
13217 posts

Uber Geek

Moderator

ID Verified

Trusted

Lifetime subscriber

#1351378 24-Jul-2015 21:24

SumnerBoy: I did look at Docker Michael, but the networking side of things looked a bit fiddly. I like the idea of lightweight containers rather than full VMs, that is why I am looking at Proxmox with OpenVZ.

I've used OpenVZ and find it works totally fine. I just have that one Windows VM otherwise I'd be using it :)

Worth just giving it a shot.

SumnerBoy

2068 posts

Uber Geek

ID Verified

Lifetime subscriber

#1351382 24-Jul-2015 21:32

You mean Docker works totally fine? I actually started out on all this by looking at Docker, but it just looked very tricky setting up containers to talk to one another. Setting up links etc, rather than just assigning an IP address and treating it like a separate host (like with openvz). Definitely still open to the idea of Docker tho.

One other consideration with Docker, if I have a single container for each app, as they suggest, then if there is an OS patch for something like *heartbleed* I would have to upgrade 30 odd containers. That sounds like a bit of a nightmare.

Mark

1653 posts

Uber Geek

#1351383 24-Jul-2015 21:42

SumnerBoy: Hey Mark - yeah it seems like the hardware requirements for Freenas are actually pretty heavy. They say min of 8GB of RAM! But there is plenty of anecdotal evidence of people happily running on 4GB.

I don't know enough about the ZFS - is it that much better?

Yup ZFS is resource hungry! If you don't go with de-duplication you can survive with 4GB .. but!

FreeNAS is not a great candidate for virtualisation, ZFS likes direct access to the disks so anything that gets between it and them can negate a lot of the benefits it brings. When set up properly it can be very good, when not setup correctly it can look good and then seem to go nuts and eat your data ;-)

This is a good read on the subject : http://www.freenas.org/whats-new/2015/02/a-complete-guide-to-freenas-hardware-design-part-i-purpose-and-best-practices.html

SumnerBoy

2068 posts

Uber Geek

ID Verified

Lifetime subscriber

#1351384 24-Jul-2015 21:45

Cheers Mark - I will read that now! One thing - I don't intend to have any virtualisation on the NAS (after thinking about Chev's solution I decided against running the NAS in a VM - seems a bit like a catch 22 - if the VM host died I wouldn't have access to my NAS to grab the backups...).

So it will just be a standalone box, with an i3 + 4GB plus a disk array. All booting/running off a USB flash driver (for the NAS OS). Just trying to determine if my existing MOBO (which I hope to use for the NAS) will handle 3 HDDs...

SumnerBoy

2068 posts

Uber Geek

ID Verified

Lifetime subscriber

#1351385 24-Jul-2015 21:58

Great article Mark - that was very helpful.

Mark

1653 posts

Uber Geek

#1351414 24-Jul-2015 23:39

SumnerBoy: Great article Mark - that was very helpful.

No probs! I'll pick your brains one day about home automation :-)