Geekzone: technology news, blogs, forums
Guest
Welcome Guest.
You haven't logged in yet. If you don't have an account you can register now.
View this topic in a long page with up to 500 replies per page Create new topic
1 | 2 | 3
1508 posts

Uber Geek
+1 received by user: 213


  Reply # 1202889 23-Dec-2014 20:23
Send private message

starting at $1000 ish without disks or much RAM. 




Try Vultr using this link and get us both some credit:

 

http://www.vultr.com/?ref=7033587-3B


760 posts

Ultimate Geek
+1 received by user: 170


  Reply # 1203970 26-Dec-2014 11:25
One person supports this post
Send private message

I use Freebsd and ZFS. I have done so for years and have never lost any data despite hardware failures, disk failures and power outages.

It is easy to setup and run - I export shares with samba.

The hardware requirements are low. Ignore all the exaggerated horror tales about gigs of RAM per terabyte of disk. I have an old core 2 duo running with 4G of memory and it is just fine. You can also ignore all the guff you get from the likes of PB tech who will try to upsell you from a Green drive to a Red one. ZFS works well on all drive types.

BTRFS is claimed to do all that ZFS can do and more but every time I looked, important features are still in beta. ZFS has been heavy duty industrial grade production ready for years.

All you would need to get going would be a cheap old desktop from trademe and the disks that you want to use. I ssh in to my server so no need for another monitor.

I chose freebsd rather than solaris because freebsd does gpt partitioning really well. You can give each gpt partition an arbitrary label and then access partitions for ZFS with the labels and you can forget about the weird and wacky /dev/sda5p3 stuff. That is also an argument for using Freebsd and not Linux for this job.

Be aware that windows can quietly lose data as it writes to a file. I tested this out with fastsum and a binfile-diff utility a few years back. Windows would happily copy a large file from one disk to another with no reported errors. Fsum would report different md5 sums and the bin diff would highlight that the destination file was the same length as the original but that the dest file contained a block of some small number of Kbytes that were set to zero! I no longer use windows to copy any files. I use either teracopy (free edition) or, over bad networks, I use a program that I wrote that will cope with network errors and outages. Tera copy does its own checksums and if I use my program, I then use fsum. I have seen these windows errors on w2k, XP and 7. I daresay that vista and 8 will be no better.

For backups, I do this:

Split the files to be backed up into four lots of equal size.

Create a rar file for each of those four lots. The rar file will be a multi volume one and it will have a small number of recovery volumes.

I will then use quickpar over all of the resultant files to generate 50% redundant recovery files.

I write all of the rar files for the first set onto disk 1 and and so on up to disk four and then I write the par2 files onto disks 5 and 6.

I can then stash the disks into 6 different offsite locations.

If I want to recover one file, I just need to fetch the one of those six disks onto which the original file was rar'ed from its location. If the disk has bad sectors, then the rar recovery volumes will deal with that. If the whole disk has failed then I need to get at least four of the remaining 5 disks and quickpar will restore a copy of the failed disk and I can extract the file that way.



760 posts

Ultimate Geek
+1 received by user: 170


  Reply # 1203992 26-Dec-2014 12:18
Send private message

Mark: Running MD5 on top of your existing filesystem is a waste of effort to be honest, if MD5 detects the error it's too late, you need those checks done by the filesystem or underneath it.

It's amazing how many enterprise environments don't even think of bit rot and just assume the storage arrays handles it, RAID does it right ?  Oh noes it don't! :-(

So running a good modern filesystem is a must if you want to make sure your TiB of data is still the same TiB of data from 6 months ago, ZFS, BTRFS and ReFS all have the capability to keep checking and rechecking, but you need to make sure the machine itself is up to spec as well (so a UPS and ECC RAM are essential options), and there is the side effect that the discs will be active nearly all the time adding up the power bill, cooling requirements and noise.



ZFS has no need of a UPS. Raid 5 has a bug - the raid 5 write hole - which means that UPS is a must for raid 5 or raid 6 if you want to avoid data loss after a power outage. ZFS was written specifically to not have that weakness.


1360 posts

Uber Geek
+1 received by user: 345


  Reply # 1204035 26-Dec-2014 13:53
Send private message

jpoc: 

ZFS has no need of a UPS. Raid 5 has a bug - the raid 5 write hole - which means that UPS is a must for raid 5 or raid 6 if you want to avoid data loss after a power outage. ZFS was written specifically to not have that weakness.



I don't agree about not needing a UPS ... data going into the server does not instantly get written to the ZIL it's in RAM for an eternity (to a computer) so having a UPS hooked up gives the server time to do a controlled destine/shutdown.

ZFS is good, but it's not the holy saviour of data storage :-)  It's more than capable for home use but you do have to make sure it has its essential tools (UPS, ECC RAM etc)  and it is resource hungry (which is why not too many consumer NAS implementations use it ... that and licensing probably) If you can't meet it's minimum requirements  you may as well just build a simpler RAID-5 or RAID-6 setup.




14138 posts

Uber Geek
+1 received by user: 2544

Trusted
Subscriber

  Reply # 1204048 26-Dec-2014 14:21
Send private message

The Mrs has agreed data integrity is a priority. Any more nas recommendations that use a good robust reliable file system? Price is significant, prefer low maintenance, device rather than pc most likely. Min 2 disks, maybe 4 would be nice.




AWS Certified Solution Architect Professional, Sysop Administrator Associate, and Developer Associate
TOGAF certified enterprise architect
Professional photographer




14138 posts

Uber Geek
+1 received by user: 2544

Trusted
Subscriber

  Reply # 1204049 26-Dec-2014 14:22
Send private message

And thanks to jpoc for the huge useful interesting post :) Sounds good but prefer simple, ie nas.




AWS Certified Solution Architect Professional, Sysop Administrator Associate, and Developer Associate
TOGAF certified enterprise architect
Professional photographer


760 posts

Ultimate Geek
+1 received by user: 170


  Reply # 1204054 26-Dec-2014 14:37
Send private message

timmmay: And thanks to jpoc for the huge useful interesting post :) Sounds good but prefer simple, ie nas.


Then use FreeNAS - it is a FreeBSD/ZFS distro that is pre-configured to be a consumer NAS. It is well supported by community forums.



14138 posts

Uber Geek
+1 received by user: 2544

Trusted
Subscriber

  Reply # 1204090 26-Dec-2014 16:23
Send private message

Sounds good but I don't want a big pc, something tiny and fit for purpose maybe easiest for me. I know you can get small pc's but a dedicated device maybe easier.




AWS Certified Solution Architect Professional, Sysop Administrator Associate, and Developer Associate
TOGAF certified enterprise architect
Professional photographer


4577 posts

Uber Geek
+1 received by user: 2078

Trusted
Subscriber

  Reply # 1204111 26-Dec-2014 17:01
Send private message

timmmay: Sounds good but I don't want a big pc, something tiny and fit for purpose maybe easiest for me. I know you can get small pc's but a dedicated device maybe easier.


I'm pretty happy with the ReadyNAS but I'm a little suspicious of both disks in it failing within a month of each other. That might be a bad batch (of disks), maybe? But it might be heat. The system seems to run relatively hot.




iPad Air + iPhone SE + 2degrees 4tw!

These comments are my own and do not represent the opinions of 2degrees.


760 posts

Ultimate Geek
+1 received by user: 170


  Reply # 1204115 26-Dec-2014 17:09
Send private message

SaltyNZ:
timmmay: Sounds good but I don't want a big pc, something tiny and fit for purpose maybe easiest for me. I know you can get small pc's but a dedicated device maybe easier.


I'm pretty happy with the ReadyNAS but I'm a little suspicious of both disks in it failing within a month of each other. That might be a bad batch (of disks), maybe? But it might be heat. The system seems to run relatively hot.


Yes, heat is a big issue in these things. small plastic cases with inadequate fans......

I have a couple of 6T drives and they run hot even in a big well vented case.


760 posts

Ultimate Geek
+1 received by user: 170


  Reply # 1204134 26-Dec-2014 17:36
Send private message

timmmay: Sounds good but I don't want a big pc, something tiny and fit for purpose maybe easiest for me. I know you can get small pc's but a dedicated device maybe easier.


In your case, I might go for something like this off trademe.

http://www.trademe.co.nz/computers/desktops/no-monitor/auction-825070257.htm

I believe that you can fit three drives in there - a boot drive and two data drives - and it will go up to 8G of RAM.

FreeNAS does not need an EFI BIOS to run with 6T disks.

Sure that is an old machine but I have never actually had an ancient Dell desktop fail on me. (Same for IBM/Lenovo.)



21447 posts

Uber Geek
+1 received by user: 4354

Trusted
Subscriber

  Reply # 1204145 26-Dec-2014 18:00
Send private message

Old machines will use more power than a new machine doing the same things so those budget ex lease machines are often a false economy for always on applications.




Richard rich.ms



14138 posts

Uber Geek
+1 received by user: 2544

Trusted
Subscriber

  Reply # 1206151 30-Dec-2014 16:58
Send private message

Based on the advice from this thread I've started a thread to ask for advice about a NAS - if anyone can contribute the thread's here :)




AWS Certified Solution Architect Professional, Sysop Administrator Associate, and Developer Associate
TOGAF certified enterprise architect
Professional photographer


700 posts

Ultimate Geek
+1 received by user: 287

Subscriber

  Reply # 1206158 30-Dec-2014 17:34
Send private message

Before you spend any money have you tried calculating the checksum using an alternative tool to verify it's not a bug in fastsum?










14138 posts

Uber Geek
+1 received by user: 2544

Trusted
Subscriber

  Reply # 1206160 30-Dec-2014 17:41
Send private message

May do that thanks.




AWS Certified Solution Architect Professional, Sysop Administrator Associate, and Developer Associate
TOGAF certified enterprise architect
Professional photographer


1 | 2 | 3
View this topic in a long page with up to 500 replies per page Create new topic

Twitter »

Follow us to receive Twitter updates when new discussions are posted in our forums:



Follow us to receive Twitter updates when news items and blogs are posted in our frontpage:



Follow us to receive Twitter updates when tech item prices are listed in our price comparison site:



Geekzone Live »

Try automatic live updates from Geekzone directly in your browser, without refreshing the page, with Geekzone Live now.



Are you subscribed to our RSS feed? You can download the latest headlines and summaries from our stories directly to your computer or smartphone by using a feed reader.

Alternatively, you can receive a daily email with Geekzone updates.