Geekzone: technology news, blogs, forums
Guest
Welcome Guest.
You haven't logged in yet. If you don't have an account you can register now.




117 posts

Master Geek
+1 received by user: 21


Topic # 185249 16-Nov-2015 01:03
3 people support this post
Send private message

When I was but a young and foolish lad, I gathered components from the local PB Tech and put them together in an unnecessarily expensive micro-atx case to create an Ubuntu based fileserver.

In a week of stumbling around I managed to mount 4x 2TB Seagate Barracuda drives into a software RAID10 array and filled them up with music, movies and all my family’s documents and photos.

I looked upon my work and saw that it was good.

Between the 7th and 1168th day I rested, safe in the knowledge that a RAID array is the best possible backup system for one’s data. On the 1169th day I tried to log in to the server remotely and update it (as I do roughly every 3 months). No response.

I couldn’t tell the server’s status, but it was still powered on, meaning that (depending when the first disk died) there could have been up to 12 weeks of the array sitting there in a degraded state, not good because degraded arrays make the other drives more likely to fail. MDADM’s warning emails had failed to trigger and the reality came crashing down: Putting your data in a RAID array isn’t backing it up, it just allows you to keep going with minimum downtime in the event of hardware failure.

Years’ worth of documents and family photos (stored in no other location because “the RAID is the backup! Durrrr!”) now hung in the balance, but fortunately, only one disk had failed and somehow (Odin be praised) the server had reset and couldn’t get past boot without some user input. It didn’t try to remount the array and the other disks survived.

With some help from a friend, I managed to remount the degraded array and copy all the data safely onto a brand new NAS box (yes with real RAID friendly drives in it this time). I’m setting up the online backup as we speak.

Had I thought this through when I built the damn thing I no doubt would have organized some sort of actual backup to an online service, I was just saved by a lucky chain of events.
But you might not be so lucky! So for critical data, remember to treat the RAID arrays in your servers or NAS boxes as ‘uptime contingencies’ and make separate backups!  


TLDR: Back your data up bro.

View this topic in a long page with up to 500 replies per page Create new topic
 1 | 2
Doesn't know what he doin
2841 posts

Uber Geek
+1 received by user: 328

Subscriber

  Reply # 1428556 16-Nov-2015 01:25
3 people support this post
Send private message

500 years in the future, backing up data will still be the lesson that everyone on this planet will learn the hard way (in varying degrees). :)




Bachelor of Computing Systems (2015)

 

--

 

Late 2013 MacBook Pro with Retina Display (4GB/2.4GHz i5/128GB SSD) - HP DV6 (8GB/2.8GHz i7/120GB SSD + 750GB HDD)
iPhone 6S + (64GB/Gold/Vodafone NZ) - Xperia Z C6603 (16GB/White/Spark NZ)

Sam, Auckland 


2577 posts

Uber Geek
+1 received by user: 372


  Reply # 1428565 16-Nov-2015 06:39
Send private message

server 101

As you have figured out RAID is not a backup. RAID helps provide seamless service if a disk fails, you still need to back up.

You're not the the first nor will you be the last to make the mistake of believing RAID is a back up.

For pretty well all stuff in the home a backup syzstem is far far more important than a RAID.




Sony Xperia X Running Sailfish
Jolla C
Nokia N1
Dell Inspiron 14z i5


 
 
 
 


251 posts

Ultimate Geek
+1 received by user: 115


  Reply # 1428572 16-Nov-2015 07:44
Send private message

I think you did well - the RAID did exactly what it was meant to do, preserve your data when their is a drive failure.

How far you go depends on the level of problem you want to mitigate, and how much money you are prepared to spend to mitigate it:
* Worried about drive failure, use RAID
* Worried about getting back an old copy of a file over-written - need snapshots or independent copy of the data
* Worried about box failure - need a complete secondary copy of data
* Worried about fire/natural disaster - need a complete secondary copy of data offsite
* Worried about restore time and a fire - need a complete secondary copy of data onsite and offsite
* Worried about site failure and restore time - need a complete secondary copy of data onsite and offsite and a DR site ready to go with access to data
And you keep going on, only limited by the amount of money and time you are prepared to spend mitigating increasingly unlikely events.




Try my latest project, a Cisco type 5 enable secret password cracker written in javascript!

1344 posts

Uber Geek
+1 received by user: 118


  Reply # 1428609 16-Nov-2015 08:45
2 people support this post
Send private message

I have to say that I am HAPPY to see someone actually use software RAID and be able to recover data, rather than the hardware raid where disks are only readable on the controller they created them on.  And that you didn't use RAID5!

I run a software RAID1 (in Windows) for the more important stuff (but I have both an on-site backup and a copy in the Cloud) and I know if a disk dies, the entire contents of the other is readable by any Windows machine.

1270 posts

Uber Geek
+1 received by user: 311


  Reply # 1428616 16-Nov-2015 09:01
One person supports this post
Send private message

Excellent thread.
TL:DR? RAID is not a backup.

Glad to hear you got your files back.

3570 posts

Uber Geek
+1 received by user: 388

Trusted

  Reply # 1428622 16-Nov-2015 09:08
One person supports this post
Send private message

I purposely don't run any sort of raid for this very reason.  Or it could almost me classed as software raid.  All my disks are joined together like jbod, but it has folder replication on  all folders meaning everything is in two places.   But the real kicker is all the drives are standard NTFS volumes.  So when one if failing I can pull it out and put in into an drive enclosure to pull as much of the data out.

Also I run lots of crashplan backups to folders and to the cloud.  As stated above (and of no use to you) raid is a drive protection feature, it's not a backup.




Previously known as psycik

NextPVR/OpenHAB: 
Gigabyte AMD A8 Brix --> Samsung LA46A650D via HDMI, NextPVR,OpenHAB with Aeotech ZWave Controller
Media:Chromecast v2, ATV4, Roku3, Raspberry PI temperature Sensors and Bluetooth LE Sensors,HDHomeRun Dual
Windows 2012 
Host (Plex Server/Crashplan): 2x2TB, 2x3TB, 1x4TB using DriveBender, Samsung 850 evo 512 GB SSD, Hyper-V Server with 1xW10, 1xW2k8, 2xUbuntu 16.04 LTS, Crashplan, NextPVR channel for Plex,NextPVR Metadata Agent and Scanner for Plex


BDFL - Memuneh
59064 posts

Uber Geek
+1 received by user: 10341

Administrator
Trusted
Geekzone
Subscriber

  Reply # 1428633 16-Nov-2015 09:26
Send private message

Incindre: Between the 7th and 1168th day I rested, safe in the knowledge that a RAID array is the best possible backup system for one’s data. 


You've learnt a valuable lesson: RAID is not a backup system

davidcole: I purposely don't run any sort of raid for this very reason.  Or it could almost me classed as software raid.  All my disks are joined together like jbod, but it has folder replication on  all folders meaning everything is in two places.   But the real kicker is all the drives are standard NTFS volumes.  So when one if failing I can pull it out and put in into an drive enclosure to pull as much of the data out.


Yep, same here. I have two 4TB ReFS drives in a mirror setup. If one dies I can still access the data. 

Incindre: TLDR: Back your data up bro.


Preferably in two different locations with two different services: external HDDs (on rotation) plus an offsite cloud service such as Crashplan or similar. One is fast, one is not prone to be destroyed on a fire.






'That VDSL Cat'
6664 posts

Uber Geek
+1 received by user: 1266

Trusted
Spark
Subscriber

  Reply # 1428645 16-Nov-2015 09:41
Send private message

RAID should never be seen as backups, but Redundancy from a failure (assuming your not using it for a RAID0 or such ofcourse..)

Personally i avoid using proper RAID, favouring software based RAID Where my data is parity protected, in the event of a failure it is recoverable atleast then - Certainly no mission critical data on this array though, apart from photos which also live in both dropbox and crashplan.

The ability to simply remove a disk and read it straight was rather compelling to me when i built the machine and only spin up the required disk when needed is nice too.




#include <std_disclaimer>

 

Any comments made are personal opinion and do not reflect directly on the position my current or past employers may have.


79 posts

Master Geek
+1 received by user: 20


Reply # 1434145 24-Nov-2015 21:33
Send private message

hio77: RAID should never be seen as backups, but Redundancy from a failure (assuming your not using it for a RAID0 or such ofcourse..)

Are you telling me that my 6 disk raid 0 array is not safe?!

533 posts

Ultimate Geek
+1 received by user: 71

Trusted
Subscriber

  Reply # 1434152 24-Nov-2015 21:44
Send private message

A valuable lesson. Unfortunately there are still people dont understand the power of the delete key (and/or shift delete). =-)

'That VDSL Cat'
6664 posts

Uber Geek
+1 received by user: 1266

Trusted
Spark
Subscriber

  Reply # 1434157 24-Nov-2015 21:52
One person supports this post
Send private message

AHitman:
hio77: RAID should never be seen as backups, but Redundancy from a failure (assuming your not using it for a RAID0 or such ofcourse..)

Are you telling me that my 6 disk raid 0 array is not safe?!


im just going to laugh and assume thats a joke :P




#include <std_disclaimer>

 

Any comments made are personal opinion and do not reflect directly on the position my current or past employers may have.


2281 posts

Uber Geek
+1 received by user: 932


  Reply # 1434171 24-Nov-2015 22:16
Send private message

So what are people using for large backup sets then?

I have about 9TB on my RAID array at the moment (RAID5 NAS) replicated to USB hard drives. Data seems to grows at 1-2TB a year. Mostly media, so once it hits the NAS it never changes, and about 100GB of data that changes frequently.

Having lost a couple of USB drives over recent years, I'm no longer too keen on USB drives as a backup option, despite using this option at present.

The two options I'm thinking of are 50GB dual-layer blurays (Verbatims can be landed for about $3.50 each = circa $70 per TB), or picking up a second hand LTO-4 tape drive.

3357 posts

Uber Geek
+1 received by user: 1776

Trusted
Spark NZ

  Reply # 1434258 25-Nov-2015 01:02
3 people support this post
Send private message

It's opportune moments like this when I break out the odd war story from a job in a previous life where I built and tested RAID systems and servers for a speciality company supplying government (including the Bureau that shall not be named, councils, universities and many ISPs). Skipping over the less relevant but still cool stories like testing controller failover with a sacrificial set of SCSI cables and a set of wirecutters, I am reminded of the ultimate "RAID is not a backup" story. Oh hell... Now I have remembered a second one... OK, I'll tell them both.

The tale of the data recovery that wasn't and never would be.

I get a call from a tech at an unnamed ISP asking about data recovery from the RAID they stored all their customer mail on. (Now this was the old days when it was all SMTP/POP, so to be fair it was basically a transient store)

"Neil, I think a disk failed, and now we can't access the drive at all... How do we fix it?" At this stage, it could have been ANYTHING so I asked some basic questions about how many flashing lights there were, and what the LCD display said. Things didn't seem to line up and ultimately it transpired that before calling me, as near as I could ascertain, they had tried moving the disks around in the chassis. This in itself wouldn't have killed them if they had done that while the array was powered down, but they had it setup as a RAID 5 with no hot spare, and one of the disks had failed at some time in the past (never found out when) and somehow they ended up pulling disks out of the degraded array while it was running. I politely explained that the data was toast and if they really wanted to ship me the drives, I might be able to force mount it and do something... Next thing I heard about it was a note on the ISP status page saying there had been a small error and they had fixed it, and that no email had been lost. Yeah right :-)

The tech with the magic touch

I was called into [same nameless ISP] to have a look at a hardware RAID controller which was working, but they had lost all management access to... Not only was the serial(!) port not responding but the LCD wasn't displaying anything although the buttons DID beep when you pressed them. After quite some mucking about and consulting a manual plus bringing a spare controller out and plugging it in I figured that someone had managed to set the LCD brightness to 0 (or whatever) and turned the display invisible! The serial port was also set to something a long way from 9600 8NE1... I ended up fixing it blind by navigating the menu on a 16*2 LCD screen using the spare controller to go keypress by keypress to the setup menu and fixing the LCD settings... Gotta love the lack of exception testing that allowed the screen to be set unreadable!

OK, remembered a third.

The news server that kept forgetting

The two items above both used good, brand name controllers and nice SCSI drives, but for the news server (at still the same unnamed ISP) they wanted something cheap and cheerful, so yep... First generation of IDE 'hardware' raid it was. That, in itself wasn't the kiss of death... It was the drive choice... The infamous IBM Deathstar 75GXP. 75GB of unreliable IDE badness. At one stage I was ferrying out a new drive every week or so to try and keep this poor machine running. I remember at one point I ordered 4-5 spares and ended up just letting them keep them in a desk drawer so they could swap them in as needed. As I recall, that array spent literally more time degraded and rebuilding than it did in a resilient state. We should have given up on those disks a lot earlier than we did, but at the time they were the new hotness and weren't known as the Deathstar at that point - and the supplier was free and easy with the advance warranty replacements.

Data integrity is a lot more than a technology. It's a philosophy. Only preparation and common sense are essential - everything else can be faked or substituted. My first computer was an Atari 600XL in 1984, and _TO THIS DAY_ I have never had a virus or lost any data. Actually, that's not strictly true - I had a CF card fail in a camera once and lost a few dozen photos (out of several hundred on the card) during the recovery attempt.

Despite have designed, built and used RAID professionally in the past, I don't use it myself and I have about 3TB of photos to archive, and [many] TB of more transient data. All the photos are backed up to a single disk NAS and to Backblaze, and everything is archived every 2-3 months onto a set of HDDs that go into a waterproof container which is then stored in a fireproof safe. (It's not actually a data safe but it's likely good enough). RAID is great in business and enterprise for continuity of data availability... But there are other ways to help with that now.

Oh...
Bonus story

The local IT guys great idea.

I had flown from Auckland to Wellington to do a hardware refresh of a significant sized database and ecommerce system and this enterprise did things very well... They had offices in Akl and Wlg with replicated databases and either centre could operate the business if the other building fell over completely.

However, when I got to the Wellington office at about 5pm on Friday and they had just switched over to the Auckland site answering all the calls and doing all the processing), I noticed a flashing light on their main RAID array so I asked the local IT guy... "Oh, that array kept failing whatever disk was in slot 3 and beeping... Whenever I replaced that disk it never even completed the rebuild and failed so I disabled the beeper"... "When did that happen?"... "Last... June, July, perhaps?"

Sharp intake of breath through teeth. Check backups. Hmm, that's odd. Tape backups haven't worked for months. Local IT person however knew exactly what had happened there. "The tape drive was way too slow, so I changed the backup destination to be drive [whatever, a high letter, like K or something], it has heaps of spare space and is much faster"... You guessed it, yes, that logical drive was on the same physical array that had been degraded for months. While their really critical stuff was replicated to Auckland, all their emails, local documents and heaps of quite important stuff had been spinning around at 15000 RPM without a safety net for months...

"So, you've got a service contract which covers all of this, why didn't you call us?"... "Didn't want to bother you"

Anyway, the next 12 odd hours - from about 7pm until 7am - were spent fixing the tape drive software and creating and testing a full backup, copying all the simple files to a separate machine and then FINALLY, at about 8am the next day, starting the hardware upgrade. It all worked out fine and there was never an interruption to service or data loss, but soooo close.

Cheers - N


14928 posts

Uber Geek
+1 received by user: 3818

Trusted
Subscriber

  Reply # 1434259 25-Nov-2015 01:04
Send private message

USB Drives are not a reliable backup method. In my experience they fail about as much as a floppy disk :) 

Better than nothing, but only JUST.

20368 posts

Uber Geek
+1 received by user: 3872

Trusted
Subscriber

  Reply # 1434271 25-Nov-2015 05:41
Send private message

USB drives fail because people move them and stuff.

I've given up on windows raid 5 now. Too often a drive will get kicked out and force a resync when you reactivate it.

Drives are fine. Just windows being a dick. Wd red so should be fine as far as timeouts etc. Now I just have them all as mirrors. That way I can pull one drive to take off site as a backup and slap a new one in to rebuild.




Richard rich.ms

 1 | 2
View this topic in a long page with up to 500 replies per page Create new topic



Twitter »

Follow us to receive Twitter updates when new discussions are posted in our forums:



Follow us to receive Twitter updates when news items and blogs are posted in our frontpage:



Follow us to receive Twitter updates when tech item prices are listed in our price comparison site:





News »

From small to medium and beyond: Navigating the ERP battlefield
Posted 21-Nov-2017 21:12


Business owners: ERP software selection starts (and finishes) with you
Posted 21-Nov-2017 21:11


Why I'm not an early adopter
Posted 21-Nov-2017 10:39


Netatmo launches smart home products in New Zealand
Posted 20-Nov-2017 20:06


Huawei Mate 10: Punchy, long battery life, artificial intelligence
Posted 20-Nov-2017 16:30


Propel launch Disney Star Wars Laser Battle Drones
Posted 19-Nov-2017 21:26


UFB killer app: Speed
Posted 17-Nov-2017 17:01


The case for RSS — MacSparky
Posted 13-Nov-2017 14:35


WordPress and Indieweb: Take control of your online presence — 6:30 GridAKL Nov 30
Posted 11-Nov-2017 13:43


Chorus reveals technology upgrade for schools, students
Posted 10-Nov-2017 10:28


Vodafone says Internet of Things (IoT) crucial for digital transformation
Posted 10-Nov-2017 10:06


Police and Facebook launch AMBER Alerts system in NZ
Posted 9-Nov-2017 10:49


Amazon debuts Fire TV Stick Basic Edition in over 100 new countries
Posted 8-Nov-2017 05:34


Vodafone VoIP transition to start this month
Posted 7-Nov-2017 12:33


Spark enhances IoT network capability
Posted 7-Nov-2017 11:33



Geekzone Live »

Try automatic live updates from Geekzone directly in your browser, without refreshing the page, with Geekzone Live now.



Are you subscribed to our RSS feed? You can download the latest headlines and summaries from our stories directly to your computer or smartphone by using a feed reader.

Alternatively, you can receive a daily email with Geekzone updates.