When your RAID dies – A cautionary tale

Forums › Desktop computing › When your RAID dies – A cautionary tale

Incindre

117 posts

Master Geek

#185249 16-Nov-2015 01:03

When I was but a young and foolish lad, I gathered components from the local PB Tech and put them together in an unnecessarily expensive micro-atx case to create an Ubuntu based fileserver.

In a week of stumbling around I managed to mount 4x 2TB Seagate Barracuda drives into a software RAID10 array and filled them up with music, movies and all my family’s documents and photos.

I looked upon my work and saw that it was good.

Between the 7th and 1168th day I rested, safe in the knowledge that a RAID array is the best possible backup system for one’s data. On the 1169th day I tried to log in to the server remotely and update it (as I do roughly every 3 months). No response.

I couldn’t tell the server’s status, but it was still powered on, meaning that (depending when the first disk died) there could have been up to 12 weeks of the array sitting there in a degraded state, not good because degraded arrays make the other drives more likely to fail. MDADM’s warning emails had failed to trigger and the reality came crashing down: Putting your data in a RAID array isn’t backing it up, it just allows you to keep going with minimum downtime in the event of hardware failure.

Years’ worth of documents and family photos (stored in no other location because “the RAID is the backup! Durrrr!”) now hung in the balance, but fortunately, only one disk had failed and somehow (Odin be praised) the server had reset and couldn’t get past boot without some user input. It didn’t try to remount the array and the other disks survived.

With some help from a friend, I managed to remount the degraded array and copy all the data safely onto a brand new NAS box (yes with real RAID friendly drives in it this time). I’m setting up the online backup as we speak.

Had I thought this through when I built the damn thing I no doubt would have organized some sort of actual backup to an online service, I was just saved by a lucky chain of events.
But you might not be so lucky! So for critical data, remember to treat the RAID arrays in your servers or NAS boxes as ‘uptime contingencies’ and make separate backups!

TLDR: Back your data up bro.

1 | 2

3075 posts

Uber Geek

#1428556 16-Nov-2015 01:25

500 years in the future, backing up data will still be the lesson that everyone on this planet will learn the hard way (in varying degrees). :)

Technofreak

6530 posts

Uber Geek

Trusted

#1428565 16-Nov-2015 06:39

server 101

As you have figured out RAID is not a backup. RAID helps provide seamless service if a disk fails, you still need to back up.

You're not the the first nor will you be the last to make the mistake of believing RAID is a back up.

For pretty well all stuff in the home a backup syzstem is far far more important than a RAID.

Sony Xperia XA2 running Sailfish OS. https://sailfishos.org The true independent open source mobile OS
Samsung Galaxy Tab S6
Dell Inspiron 14z i5

pdath

252 posts

Ultimate Geek

#1428572 16-Nov-2015 07:44

I think you did well - the RAID did exactly what it was meant to do, preserve your data when their is a drive failure.

How far you go depends on the level of problem you want to mitigate, and how much money you are prepared to spend to mitigate it:
* Worried about drive failure, use RAID
* Worried about getting back an old copy of a file over-written - need snapshots or independent copy of the data
* Worried about box failure - need a complete secondary copy of data
* Worried about fire/natural disaster - need a complete secondary copy of data offsite
* Worried about restore time and a fire - need a complete secondary copy of data onsite and offsite
* Worried about site failure and restore time - need a complete secondary copy of data onsite and offsite and a DR site ready to go with access to data
And you keep going on, only limited by the amount of money and time you are prepared to spend mitigating increasingly unlikely events.

Try my latest project, a Cisco type 5 enable secret password cracker written in javascript!

timbosan

2159 posts

Uber Geek

#1428609 16-Nov-2015 08:45

I have to say that I am HAPPY to see someone actually use software RAID and be able to recover data, rather than the hardware raid where disks are only readable on the controller they created them on. And that you didn't use RAID5!

I run a software RAID1 (in Windows) for the more important stuff (but I have both an on-site backup and a copy in the Cloud) and I know if a disk dies, the entire contents of the other is readable by any Windows machine.

Dairyxox

1594 posts

Uber Geek

#1428616 16-Nov-2015 09:01

Excellent thread.
TL:DR? RAID is not a backup.

Glad to hear you got your files back.

davidcole

6034 posts

Uber Geek

Trusted

#1428622 16-Nov-2015 09:08

I purposely don't run any sort of raid for this very reason. Or it could almost me classed as software raid. All my disks are joined together like jbod, but it has folder replication on all folders meaning everything is in two places. But the real kicker is all the drives are standard NTFS volumes. So when one if failing I can pull it out and put in into an drive enclosure to pull as much of the data out.

Also I run lots of crashplan backups to folders and to the cloud. As stated above (and of no use to you) raid is a drive protection feature, it's not a backup.

Previously known as psycik

Home Assistant: Gigabyte AMD A8 Brix, Home Assistant with Aeotech ZWave Controller, Raspberry PI, Wemos D1 Mini, Zwave, Shelly Humidity and Temperature sensors
Media:Chromecast v2, ATV4 4k, ATV4, HDHomeRun Dual
Server Host Plex Server 3x3TB, 4x4TB using MergerFS, Samsung 850 evo 512 GB SSD, Proxmox Server with 1xW10, 2xUbuntu 22.04 LTS, Backblaze Backups, usenetprime.com fastmail.com Sharesies Trakt.TV Sharesight

freitasm

BDFL - Memuneh
79283 posts

Uber Geek

Administrator

ID Verified

Trusted

Geekzone

Lifetime subscriber

#1428633 16-Nov-2015 09:26

Incindre: Between the 7th and 1168th day I rested, safe in the knowledge that a RAID array is the best possible backup system for one’s data.

You've learnt a valuable lesson: RAID is not a backup system

davidcole: I purposely don't run any sort of raid for this very reason. Or it could almost me classed as software raid. All my disks are joined together like jbod, but it has folder replication on all folders meaning everything is in two places. But the real kicker is all the drives are standard NTFS volumes. So when one if failing I can pull it out and put in into an drive enclosure to pull as much of the data out.

Yep, same here. I have two 4TB ReFS drives in a mirror setup. If one dies I can still access the data.

Incindre: TLDR: Back your data up bro.

Preferably in two different locations with two different services: external HDDs (on rotation) plus an offsite cloud service such as Crashplan or similar. One is fast, one is not prone to be destroyed on a fire.

Tradepub

Free professional, reference and technical white papers (affiliate link).

hio77

12999 posts

Uber Geek

ID Verified

Trusted

Lizard Networks

#1428645 16-Nov-2015 09:41

RAID should never be seen as backups, but Redundancy from a failure (assuming your not using it for a RAID0 or such ofcourse..)

Personally i avoid using proper RAID, favouring software based RAID Where my data is parity protected, in the event of a failure it is recoverable atleast then - Certainly no mission critical data on this array though, apart from photos which also live in both dropbox and crashplan.

The ability to simply remove a disk and read it straight was rather compelling to me when i built the machine and only spin up the required disk when needed is nice too.

#include <std_disclaimer>

Any comments made are personal opinion and do not reflect directly on the position my current or past employers may have.

AHitman

85 posts

Master Geek

#1434145 24-Nov-2015 21:33

hio77: RAID should never be seen as backups, but Redundancy from a failure (assuming your not using it for a RAID0 or such ofcourse..)

Are you telling me that my 6 disk raid 0 array is not safe?!

AKLWestie

641 posts

Ultimate Geek

Trusted

Lifetime subscriber

#1434152 24-Nov-2015 21:44

A valuable lesson. Unfortunately there are still people dont understand the power of the delete key (and/or shift delete). =-)

hio77

12999 posts

Uber Geek

ID Verified

Trusted

Lizard Networks

#1434157 24-Nov-2015 21:52

AHitman:
hio77: RAID should never be seen as backups, but Redundancy from a failure (assuming your not using it for a RAID0 or such ofcourse..)

Are you telling me that my 6 disk raid 0 array is not safe?!

im just going to laugh and assume thats a joke :P

#include <std_disclaimer>

Any comments made are personal opinion and do not reflect directly on the position my current or past employers may have.

JimmyH

2886 posts

Uber Geek

#1434171 24-Nov-2015 22:16

So what are people using for large backup sets then?

I have about 9TB on my RAID array at the moment (RAID5 NAS) replicated to USB hard drives. Data seems to grows at 1-2TB a year. Mostly media, so once it hits the NAS it never changes, and about 100GB of data that changes frequently.

Having lost a couple of USB drives over recent years, I'm no longer too keen on USB drives as a backup option, despite using this option at present.

The two options I'm thinking of are 50GB dual-layer blurays (Verbatims can be landed for about $3.50 each = circa $70 per TB), or picking up a second hand LTO-4 tape drive.

Talkiet

4792 posts

Uber Geek

Trusted

#1434258 25-Nov-2015 01:02

It's opportune moments like this when I break out the odd war story from a job in a previous life where I built and tested RAID systems and servers for a speciality company supplying government (including the Bureau that shall not be named, councils, universities and many ISPs). Skipping over the less relevant but still cool stories like testing controller failover with a sacrificial set of SCSI cables and a set of wirecutters, I am reminded of the ultimate "RAID is not a backup" story. Oh hell... Now I have remembered a second one... OK, I'll tell them both.

The tale of the data recovery that wasn't and never would be.

I get a call from a tech at an unnamed ISP asking about data recovery from the RAID they stored all their customer mail on. (Now this was the old days when it was all SMTP/POP, so to be fair it was basically a transient store)

"Neil, I think a disk failed, and now we can't access the drive at all... How do we fix it?" At this stage, it could have been ANYTHING so I asked some basic questions about how many flashing lights there were, and what the LCD display said. Things didn't seem to line up and ultimately it transpired that before calling me, as near as I could ascertain, they had tried moving the disks around in the chassis. This in itself wouldn't have killed them if they had done that while the array was powered down, but they had it setup as a RAID 5 with no hot spare, and one of the disks had failed at some time in the past (never found out when) and somehow they ended up pulling disks out of the degraded array while it was running. I politely explained that the data was toast and if they really wanted to ship me the drives, I might be able to force mount it and do something... Next thing I heard about it was a note on the ISP status page saying there had been a small error and they had fixed it, and that no email had been lost. Yeah right :-)

The tech with the magic touch

I was called into [same nameless ISP] to have a look at a hardware RAID controller which was working, but they had lost all management access to... Not only was the serial(!) port not responding but the LCD wasn't displaying anything although the buttons DID beep when you pressed them. After quite some mucking about and consulting a manual plus bringing a spare controller out and plugging it in I figured that someone had managed to set the LCD brightness to 0 (or whatever) and turned the display invisible! The serial port was also set to something a long way from 9600 8NE1... I ended up fixing it blind by navigating the menu on a 16*2 LCD screen using the spare controller to go keypress by keypress to the setup menu and fixing the LCD settings... Gotta love the lack of exception testing that allowed the screen to be set unreadable!

OK, remembered a third.

The news server that kept forgetting

The two items above both used good, brand name controllers and nice SCSI drives, but for the news server (at still the same unnamed ISP) they wanted something cheap and cheerful, so yep... First generation of IDE 'hardware' raid it was. That, in itself wasn't the kiss of death... It was the drive choice... The infamous IBM Deathstar 75GXP. 75GB of unreliable IDE badness. At one stage I was ferrying out a new drive every week or so to try and keep this poor machine running. I remember at one point I ordered 4-5 spares and ended up just letting them keep them in a desk drawer so they could swap them in as needed. As I recall, that array spent literally more time degraded and rebuilding than it did in a resilient state. We should have given up on those disks a lot earlier than we did, but at the time they were the new hotness and weren't known as the Deathstar at that point - and the supplier was free and easy with the advance warranty replacements.

Data integrity is a lot more than a technology. It's a philosophy. Only preparation and common sense are essential - everything else can be faked or substituted. My first computer was an Atari 600XL in 1984, and _TO THIS DAY_ I have never had a virus or lost any data. Actually, that's not strictly true - I had a CF card fail in a camera once and lost a few dozen photos (out of several hundred on the card) during the recovery attempt.

Despite have designed, built and used RAID professionally in the past, I don't use it myself and I have about 3TB of photos to archive, and [many] TB of more transient data. All the photos are backed up to a single disk NAS and to Backblaze, and everything is archived every 2-3 months onto a set of HDDs that go into a waterproof container which is then stored in a fireproof safe. (It's not actually a data safe but it's likely good enough). RAID is great in business and enterprise for continuity of data availability... But there are other ways to help with that now.

Oh...
Bonus story

The local IT guys great idea.

I had flown from Auckland to Wellington to do a hardware refresh of a significant sized database and ecommerce system and this enterprise did things very well... They had offices in Akl and Wlg with replicated databases and either centre could operate the business if the other building fell over completely.

However, when I got to the Wellington office at about 5pm on Friday and they had just switched over to the Auckland site answering all the calls and doing all the processing), I noticed a flashing light on their main RAID array so I asked the local IT guy... "Oh, that array kept failing whatever disk was in slot 3 and beeping... Whenever I replaced that disk it never even completed the rebuild and failed so I disabled the beeper"... "When did that happen?"... "Last... June, July, perhaps?"

Sharp intake of breath through teeth. Check backups. Hmm, that's odd. Tape backups haven't worked for months. Local IT person however knew exactly what had happened there. "The tape drive was way too slow, so I changed the backup destination to be drive [whatever, a high letter, like K or something], it has heaps of spare space and is much faster"... You guessed it, yes, that logical drive was on the same physical array that had been degraded for months. While their really critical stuff was replicated to Auckland, all their emails, local documents and heaps of quite important stuff had been spinning around at 15000 RPM without a safety net for months...

"So, you've got a service contract which covers all of this, why didn't you call us?"... "Didn't want to bother you"

Anyway, the next 12 odd hours - from about 7pm until 7am - were spent fixing the tape drive software and creating and testing a full backup, copying all the simple files to a separate machine and then FINALLY, at about 8am the next day, starting the hardware upgrade. It all worked out fine and there was never an interruption to service or data loss, but soooo close.

Cheers - N

Please note all comments are from my own brain and don't necessarily represent the position or opinions of my employer, previous employers, colleagues, friends or pets.

networkn

Networkn
32351 posts

Uber Geek

ID Verified

Trusted

Lifetime subscriber

#1434259 25-Nov-2015 01:04

USB Drives are not a reliable backup method. In my experience they fail about as much as a floppy disk :)

Better than nothing, but only JUST.

richms

28177 posts

Uber Geek

Trusted

Lifetime subscriber

#1434271 25-Nov-2015 05:41

USB drives fail because people move them and stuff.

I've given up on windows raid 5 now. Too often a drive will get kicked out and force a resync when you reactivate it.

Drives are fine. Just windows being a dick. Wd red so should be fine as far as timeouts etc. Now I just have them all as mirrors. That way I can pull one drive to take off site as a backup and slap a new one in to rebuild.

Richard rich.ms

1 | 2