Disk encoding to combat failed sectors

Forums › Off topic › Disk encoding to combat failed sectors - aka single disk "RAID"

1 | 2 | 3

1656 posts

Uber Geek
+1 received by user: 555

#958947 30-Dec-2013 10:29

Take a look at Windows 8/2012, it has Storage Spaces and Resilient Filesystem which will more or less do what you want.

timmmay

20923 posts

Uber Geek
+1 received by user: 5395

Trusted

Lifetime subscriber

#958960 30-Dec-2013 11:13

Mark: Take a look at Windows 8/2012, it has Storage Spaces and Resilient Filesystem which will more or less do what you want.

A quick read suggest they're pretty much exactly what I'm after, thanks! I take it Resilient Filesystem's are just on the server? That W8 pro you'd just use storage spaces?

Mark

1656 posts

Uber Geek
+1 received by user: 555

#958964 30-Dec-2013 11:20

I think ReFS is in Windows 8.1 but not in 8. I'm a storage admin type and not a Windows admin type I'm afraid, hopefully one of the Windows people can help you more :-)

Now if you wanted to go full bore and put in a proper resilient storage infrastructure then I'm your man! :-)

freitasm

BDFL - Memuneh
80944 posts

Uber Geek
+1 received by user: 41698

Administrator

ID Verified

Trusted

Geekzone

Lifetime subscriber

#958979 30-Dec-2013 11:40

Windows 8.1 doesn't support format ReFS out of the box (it reads it), but you can change a registry key to allow you to format drives with this file system.

Referral links: Quic Broadband (free setup code: R587125ERQ6VE) | Samsung | AliExpress | Wise | Sharesies

Support Geekzone by subscribing (browse ads-free), or making a one-off or recurring donation through PressPatron.

freitasm

BDFL - Memuneh
80944 posts

Uber Geek
+1 received by user: 41698

Administrator

ID Verified

Trusted

Geekzone

Lifetime subscriber

#958989 30-Dec-2013 11:45

This reminds me... I have 5 TB on my HP Microserver, might move things around and format all those drives with ReFS now.

Referral links: Quic Broadband (free setup code: R587125ERQ6VE) | Samsung | AliExpress | Wise | Sharesies

Support Geekzone by subscribing (browse ads-free), or making a one-off or recurring donation through PressPatron.

Zeon

3927 posts

Uber Geek
+1 received by user: 759

Trusted

#958990 30-Dec-2013 11:47

Yea its gonna become a bigger problem. It's pretty much a killer now that pushes large RAID5s into history. Imagine if you had a 20TB RAID 5 and recovering from a drive failure - almost certainly gonna get this.

Speedtest 2019-10-14

Dyson

Shop now for Dyson appliances (affiliate link).

freitasm

BDFL - Memuneh
80944 posts

Uber Geek
+1 received by user: 41698

Administrator

ID Verified

Trusted

Geekzone

Lifetime subscriber

#959007 30-Dec-2013 11:59

Actually, in statistical terms a RAID 5 volume can cause more problems as the drives grow larger (citation needed). In other words, for a consumer level setup I'd go with a mirror at max, with good backup strategy (either NAS, online backup, manual offsite, tape).

In Tim's case (I know his house) I would say a good strategy would be a NAS in one side of the house, away from the small office in the back. If possible with a gigabit wired connection in between them. Install some software to make the folders backup (not sync!) on a timely basis to that NAS.

The volume of data to backup causes problems for true offsite online backup, so I'd suggest investing on a tape drive and routinely copy everything to tape and store away at someone else's place.

Referral links: Quic Broadband (free setup code: R587125ERQ6VE) | Samsung | AliExpress | Wise | Sharesies

Support Geekzone by subscribing (browse ads-free), or making a one-off or recurring donation through PressPatron.

Mark

1656 posts

Uber Geek
+1 received by user: 555

#959009 30-Dec-2013 12:00

Zeon: Yea its gonna become a bigger problem. It's pretty much a killer now that pushes large RAID5s into history. Imagine if you had a 20TB RAID 5 and recovering from a drive failure - almost certainly gonna get this.

Eh ? Care to explain the logic behind this ?

EDIT: Ahh putting it in context with a previous post helps :-)

RAID-5 is still around for a while yet, write penalty is lower than RAID-6 (4 vs 6 IOPs), sequential writes are better on RAID-5 (and 6) than on RAID-1 (or 10) .. all depends on what the workload is to be.

timmmay

20923 posts

Uber Geek
+1 received by user: 5395

Trusted

Lifetime subscriber

#959201 30-Dec-2013 17:54

ReFS/SS could be compelling features. I'll do a bit more research first, to make sure ReFS and Storage Spaces will give me what I want - basically better reliability. It looks like ReFS helps a little, but Storage Spaces provide a RAID like function. It looks like you need at least two disks so it can keep a mirror, which honestly is only mildly helpful. Or have I not understood it incorrectly?

I've heard some very bad things about W8... can I use 8.1 like Windows 7 pro, no touch screen, with a start menu etc? Most of my software I launch from icons on my task bar, the rest I hit the start button, type the name, and go. Basically I want Windows 7 with the technical improvements.

I'm also going to start using checksum software, Corz Checksum software seems easy and does SHA1 hashes. That will give me some piece of mind in the short term.

jpoc

1043 posts

Uber Geek
+1 received by user: 289

#959565 31-Dec-2013 13:09

It just occurred to me that nobody thought to mention virtual machines.

Could you install linux on your desktop machine with all your disks formatted with btrfs and then run windows 7 as a virtual machine under linux. Then you could continue to do all of your work as now, under win 7 but all of your disks would be protected by whatever btrfs replication schemes you selected.

freitasm

BDFL - Memuneh
80944 posts

Uber Geek
+1 received by user: 41698

Administrator

ID Verified

Trusted

Geekzone

Lifetime subscriber

#959573 31-Dec-2013 13:34

By the way, just formated all my data disk (except system disk) as ReFS on my Windows Server and recreated the mirrors.

Mirrors + Resilient File System...

Referral links: Quic Broadband (free setup code: R587125ERQ6VE) | Samsung | AliExpress | Wise | Sharesies

Support Geekzone by subscribing (browse ads-free), or making a one-off or recurring donation through PressPatron.

Geekzone support

Support Geekzone with one-off or recurring donations Donate via PressPatron.

freitasm

BDFL - Memuneh
80944 posts

Uber Geek
+1 received by user: 41698

Administrator

ID Verified

Trusted

Geekzone

Lifetime subscriber

#959574 31-Dec-2013 13:37

timmmay: Somewhere between 2TB and 4TB, but I don't back up everything to every location.

What about my suggestion of having a NAS with mirrored 4TB drives for your backup instead of playing with Windows OS?

I have a box here that would fit, just need the drives.

Referral links: Quic Broadband (free setup code: R587125ERQ6VE) | Samsung | AliExpress | Wise | Sharesies

Support Geekzone by subscribing (browse ads-free), or making a one-off or recurring donation through PressPatron.

blakamin

4431 posts

Uber Geek
+1 received by user: 1306
Inactive user

#959584 31-Dec-2013 14:22

timmmay:
can I use 8.1 like Windows 7 pro, no touch screen, with a start menu etc? Most of my software I launch from icons on my task bar, the rest I hit the start button, type the name, and go. Basically I want Windows 7 with the technical improvements.

Yup!
I run 8.1 with stardock "Start8". Looks exactly like 7, boots to desktop, and I've never seen the metro UI or whatever they call it these days.

JimmyH

2899 posts

Uber Geek
+1 received by user: 1554

#959591 31-Dec-2013 14:37

freitasm: Actually, in statistical terms a RAID 5 volume can cause more problems as the drives grow larger (citation needed). In other words, for a consumer level setup I'd go with a mirror at max, with good backup strategy (either NAS, online backup, manual offsite, tape).

In Tim's case (I know his house) I would say a good strategy would be a NAS in one side of the house, away from the small office in the back. If possible with a gigabit wired connection in between them. Install some software to make the folders backup (not sync!) on a timely basis to that NAS.

The volume of data to backup causes problems for true offsite online backup, so I'd suggest investing on a tape drive and routinely copy everything to tape and store away at someone else's place.

I have been looking at tape for exactly that reason. I had been using DVD-Rs for backup, but into multi-TB territory that starts to become infeasible. I had a drive fail, and have nearly 2TB of files to copy back (circa 400 DVDs) - which I'm not looking forward to.

However, good tape drives (LTO4 or 5) are really expensive, and you seem to need to spec the machine doing the backups to quite a high level (SAS controller, fast RAID to sustain the data transfer rate etc) otherwise the tapes will "shoeshine" and be ruined if the data rate can't keep up etc.

Most of my stuff will be created once then won't change (movie encodes on my NAS etc). So I think I will go with a dual backup strategy. I will use USB hard-drives for the first copy. This copy will be kept onsite in case of raid failure. For the second I think I will be going with 50GB Blu Ray disks. I can land quality ones for $3.60 each in quantity. It's cost competitive with tape, scalable, and at only 20 disks per TB burning and storing the disks is feasible. $72 per TB isn't ludicrously expensive.

For my small data directory (emails, photos etc) the data is usually under 20GB, so I will probably use USB drives in rotation. Possibly just 2-3 32GB flash drives.

timmmay

20923 posts

Uber Geek
+1 received by user: 5395

Trusted

Lifetime subscriber

#959600 31-Dec-2013 15:10

jpoc: It just occurred to me that nobody thought to mention virtual machines.

Could you install linux on your desktop machine with all your disks formatted with btrfs and then run windows 7 as a virtual machine under linux. Then you could continue to do all of your work as now, under win 7 but all of your disks would be protected by whatever btrfs replication schemes you selected.

Too much hassle, thanks for the idea though.

freitasm:
timmmay: Somewhere between 2TB and 4TB, but I don't back up everything to every location.

What about my suggestion of having a NAS with mirrored 4TB drives for your backup instead of playing with Windows OS?

I have a box here that would fit, just need the drives.

Thanks for the thought, and the offer. A NAS would work for onsite disks, but I'm primarily concerned with my offsite backups.

My current plan is to get one of these dual disk drive enclosures and run them RAID1 with 2x4TB WD Red. They're compact enough to move around. When I get around to doing Windows 8.1 I can used it as JBOD and use ReFS and Storage Spaces.

freitasm: By the way, just formated all my data disk (except system disk) as ReFS on my Windows Server and recreated the mirrors.

Mirrors + Resilient File System...

My reading suggests you need to use Storage Spaces if you want error correction. ReFS will give you error detection with checksums, but not error correction.

I've started using a little program called Corz for validation, creating SH1 hashes of directories. The problem I'm running into is immediately after creating hashes on 1TB of data some files/directories are failing validation. I'm switching from SHA1 to MD5 to see if it's just a bug with the program.

1 | 2 | 3