Geekzone: technology news, blogs, forums
Guest
Welcome Guest.
You haven't logged in yet. If you don't have an account you can register now.




117 posts

Master Geek
+1 received by user: 21


Topic # 185249 16-Nov-2015 01:03
3 people support this post
Send private message

When I was but a young and foolish lad, I gathered components from the local PB Tech and put them together in an unnecessarily expensive micro-atx case to create an Ubuntu based fileserver.

In a week of stumbling around I managed to mount 4x 2TB Seagate Barracuda drives into a software RAID10 array and filled them up with music, movies and all my family’s documents and photos.

I looked upon my work and saw that it was good.

Between the 7th and 1168th day I rested, safe in the knowledge that a RAID array is the best possible backup system for one’s data. On the 1169th day I tried to log in to the server remotely and update it (as I do roughly every 3 months). No response.

I couldn’t tell the server’s status, but it was still powered on, meaning that (depending when the first disk died) there could have been up to 12 weeks of the array sitting there in a degraded state, not good because degraded arrays make the other drives more likely to fail. MDADM’s warning emails had failed to trigger and the reality came crashing down: Putting your data in a RAID array isn’t backing it up, it just allows you to keep going with minimum downtime in the event of hardware failure.

Years’ worth of documents and family photos (stored in no other location because “the RAID is the backup! Durrrr!”) now hung in the balance, but fortunately, only one disk had failed and somehow (Odin be praised) the server had reset and couldn’t get past boot without some user input. It didn’t try to remount the array and the other disks survived.

With some help from a friend, I managed to remount the degraded array and copy all the data safely onto a brand new NAS box (yes with real RAID friendly drives in it this time). I’m setting up the online backup as we speak.

Had I thought this through when I built the damn thing I no doubt would have organized some sort of actual backup to an online service, I was just saved by a lucky chain of events.
But you might not be so lucky! So for critical data, remember to treat the RAID arrays in your servers or NAS boxes as ‘uptime contingencies’ and make separate backups!  


TLDR: Back your data up bro.

View this topic in a long page with up to 500 replies per page Create new topic
 1 | 2
Doesn't know what he doin
2821 posts

Uber Geek
+1 received by user: 324

Subscriber

  Reply # 1428556 16-Nov-2015 01:25
3 people support this post
Send private message

500 years in the future, backing up data will still be the lesson that everyone on this planet will learn the hard way (in varying degrees). :)




Bachelor of Computing Systems (2015)

 

--

 

Late 2013 MacBook Pro with Retina Display (4GB/2.4GHz i5/128GB SSD) - HP DV6 (8GB/2.8GHz i7/120GB SSD + 750GB HDD)
iPhone 6S + (64GB/Gold/Vodafone NZ) - Xperia Z C6603 (16GB/White/Spark NZ)

Sam, Auckland 


2563 posts

Uber Geek
+1 received by user: 368


  Reply # 1428565 16-Nov-2015 06:39
Send private message

server 101

As you have figured out RAID is not a backup. RAID helps provide seamless service if a disk fails, you still need to back up.

You're not the the first nor will you be the last to make the mistake of believing RAID is a back up.

For pretty well all stuff in the home a backup syzstem is far far more important than a RAID.




Jolla C
Nokia N1
Nokia N9
Nokia E7
HP Touchpad
Dell Inspiron 14z i5


 
 
 
 


251 posts

Ultimate Geek
+1 received by user: 110


  Reply # 1428572 16-Nov-2015 07:44
Send private message

I think you did well - the RAID did exactly what it was meant to do, preserve your data when their is a drive failure.

How far you go depends on the level of problem you want to mitigate, and how much money you are prepared to spend to mitigate it:
* Worried about drive failure, use RAID
* Worried about getting back an old copy of a file over-written - need snapshots or independent copy of the data
* Worried about box failure - need a complete secondary copy of data
* Worried about fire/natural disaster - need a complete secondary copy of data offsite
* Worried about restore time and a fire - need a complete secondary copy of data onsite and offsite
* Worried about site failure and restore time - need a complete secondary copy of data onsite and offsite and a DR site ready to go with access to data
And you keep going on, only limited by the amount of money and time you are prepared to spend mitigating increasingly unlikely events.




Try my latest project, a Cisco type 5 enable secret password cracker written in javascript!

1312 posts

Uber Geek
+1 received by user: 118


  Reply # 1428609 16-Nov-2015 08:45
2 people support this post
Send private message

I have to say that I am HAPPY to see someone actually use software RAID and be able to recover data, rather than the hardware raid where disks are only readable on the controller they created them on.  And that you didn't use RAID5!

I run a software RAID1 (in Windows) for the more important stuff (but I have both an on-site backup and a copy in the Cloud) and I know if a disk dies, the entire contents of the other is readable by any Windows machine.

1249 posts

Uber Geek
+1 received by user: 301


  Reply # 1428616 16-Nov-2015 09:01
One person supports this post
Send private message

Excellent thread.
TL:DR? RAID is not a backup.

Glad to hear you got your files back.

3479 posts

Uber Geek
+1 received by user: 344

Trusted

  Reply # 1428622 16-Nov-2015 09:08
One person supports this post
Send private message

I purposely don't run any sort of raid for this very reason.  Or it could almost me classed as software raid.  All my disks are joined together like jbod, but it has folder replication on  all folders meaning everything is in two places.   But the real kicker is all the drives are standard NTFS volumes.  So when one if failing I can pull it out and put in into an drive enclosure to pull as much of the data out.

Also I run lots of crashplan backups to folders and to the cloud.  As stated above (and of no use to you) raid is a drive protection feature, it's not a backup.




Previously known as psycik

NextPVR/OpenHAB: 
Gigabyte AMD A8 Brix --> Samsung LA46A650D via HDMI, NextPVR,OpenHAB with Aeotech ZWave Controller
Media:Chromecast v2, ATV4, Roku3, Raspberry PI temperature Sensors and Bluetooth LE Sensors,HDHomeRun Dual
Windows 2012 
Host (Plex Server/Crashplan): 2x2TB, 2x3TB, 1x4TB using DriveBender, Samsung 850 evo 512 GB SSD, Hyper-V Server with 1xW10, 1xW2k8, 2xUbuntu 16.04 LTS, Crashplan, NextPVR channel for Plex,NextPVR Metadata Agent and Scanner for Plex


BDFL - Memuneh
58769 posts

Uber Geek
+1 received by user: 10171

Administrator
Trusted
Geekzone
Subscriber

  Reply # 1428633 16-Nov-2015 09:26
Send private message

Incindre: Between the 7th and 1168th day I rested, safe in the knowledge that a RAID array is the best possible backup system for one’s data. 


You've learnt a valuable lesson: RAID is not a backup system

davidcole: I purposely don't run any sort of raid for this very reason.  Or it could almost me classed as software raid.  All my disks are joined together like jbod, but it has folder replication on  all folders meaning everything is in two places.   But the real kicker is all the drives are standard NTFS volumes.  So when one if failing I can pull it out and put in into an drive enclosure to pull as much of the data out.


Yep, same here. I have two 4TB ReFS drives in a mirror setup. If one dies I can still access the data. 

Incindre: TLDR: Back your data up bro.


Preferably in two different locations with two different services: external HDDs (on rotation) plus an offsite cloud service such as Crashplan or similar. One is fast, one is not prone to be destroyed on a fire.






'That VDSL Cat'
6431 posts

Uber Geek
+1 received by user: 1230

Trusted
Spark
Subscriber

  Reply # 1428645 16-Nov-2015 09:41
Send private message

RAID should never be seen as backups, but Redundancy from a failure (assuming your not using it for a RAID0 or such ofcourse..)

Personally i avoid using proper RAID, favouring software based RAID Where my data is parity protected, in the event of a failure it is recoverable atleast then - Certainly no mission critical data on this array though, apart from photos which also live in both dropbox and crashplan.

The ability to simply remove a disk and read it straight was rather compelling to me when i built the machine and only spin up the required disk when needed is nice too.




#include <std_disclaimer>

 

Any comments made are personal opinion and do not reflect directly on the position my current or past employers may have.


79 posts

Master Geek
+1 received by user: 20


Reply # 1434145 24-Nov-2015 21:33
Send private message

hio77: RAID should never be seen as backups, but Redundancy from a failure (assuming your not using it for a RAID0 or such ofcourse..)

Are you telling me that my 6 disk raid 0 array is not safe?!

524 posts

Ultimate Geek
+1 received by user: 70

Trusted
Subscriber

  Reply # 1434152 24-Nov-2015 21:44
Send private message

A valuable lesson. Unfortunately there are still people dont understand the power of the delete key (and/or shift delete). =-)

'That VDSL Cat'
6431 posts

Uber Geek
+1 received by user: 1230

Trusted
Spark
Subscriber

  Reply # 1434157 24-Nov-2015 21:52
One person supports this post
Send private message

AHitman:
hio77: RAID should never be seen as backups, but Redundancy from a failure (assuming your not using it for a RAID0 or such ofcourse..)

Are you telling me that my 6 disk raid 0 array is not safe?!


im just going to laugh and assume thats a joke :P




#include <std_disclaimer>

 

Any comments made are personal opinion and do not reflect directly on the position my current or past employers may have.


2246 posts

Uber Geek
+1 received by user: 896


  Reply # 1434171 24-Nov-2015 22:16
Send private message

So what are people using for large backup sets then?

I have about 9TB on my RAID array at the moment (RAID5 NAS) replicated to USB hard drives. Data seems to grows at 1-2TB a year. Mostly media, so once it hits the NAS it never changes, and about 100GB of data that changes frequently.

Having lost a couple of USB drives over recent years, I'm no longer too keen on USB drives as a backup option, despite using this option at present.

The two options I'm thinking of are 50GB dual-layer blurays (Verbatims can be landed for about $3.50 each = circa $70 per TB), or picking up a second hand LTO-4 tape drive.

3304 posts

Uber Geek
+1 received by user: 1749

Trusted
Spark NZ

  Reply # 1434258 25-Nov-2015 01:02
3 people support this post
Send private message

It's opportune moments like this when I break out the odd war story from a job in a previous life where I built and tested RAID systems and servers for a speciality company supplying government (including the Bureau that shall not be named, councils, universities and many ISPs). Skipping over the less relevant but still cool stories like testing controller failover with a sacrificial set of SCSI cables and a set of wirecutters, I am reminded of the ultimate "RAID is not a backup" story. Oh hell... Now I have remembered a second one... OK, I'll tell them both.

The tale of the data recovery that wasn't and never would be.

I get a call from a tech at an unnamed ISP asking about data recovery from the RAID they stored all their customer mail on. (Now this was the old days when it was all SMTP/POP, so to be fair it was basically a transient store)

"Neil, I think a disk failed, and now we can't access the drive at all... How do we fix it?" At this stage, it could have been ANYTHING so I asked some basic questions about how many flashing lights there were, and what the LCD display said. Things didn't seem to line up and ultimately it transpired that before calling me, as near as I could ascertain, they had tried moving the disks around in the chassis. This in itself wouldn't have killed them if they had done that while the array was powered down, but they had it setup as a RAID 5 with no hot spare, and one of the disks had failed at some time in the past (never found out when) and somehow they ended up pulling disks out of the degraded array while it was running. I politely explained that the data was toast and if they really wanted to ship me the drives, I might be able to force mount it and do something... Next thing I heard about it was a note on the ISP status page saying there had been a small error and they had fixed it, and that no email had been lost. Yeah right :-)

The tech with the magic touch

I was called into [same nameless ISP] to have a look at a hardware RAID controller which was working, but they had lost all management access to... Not only was the serial(!) port not responding but the LCD wasn't displaying anything although the buttons DID beep when you pressed them. After quite some mucking about and consulting a manual plus bringing a spare controller out and plugging it in I figured that someone had managed to set the LCD brightness to 0 (or whatever) and turned the display invisible! The serial port was also set to something a long way from 9600 8NE1... I ended up fixing it blind by navigating the menu on a 16*2 LCD screen using the spare controller to go keypress by keypress to the setup menu and fixing the LCD settings... Gotta love the lack of exception testing that allowed the screen to be set unreadable!

OK, remembered a third.

The news server that kept forgetting

The two items above both used good, brand name controllers and nice SCSI drives, but for the news server (at still the same unnamed ISP) they wanted something cheap and cheerful, so yep... First generation of IDE 'hardware' raid it was. That, in itself wasn't the kiss of death... It was the drive choice... The infamous IBM Deathstar 75GXP. 75GB of unreliable IDE badness. At one stage I was ferrying out a new drive every week or so to try and keep this poor machine running. I remember at one point I ordered 4-5 spares and ended up just letting them keep them in a desk drawer so they could swap them in as needed. As I recall, that array spent literally more time degraded and rebuilding than it did in a resilient state. We should have given up on those disks a lot earlier than we did, but at the time they were the new hotness and weren't known as the Deathstar at that point - and the supplier was free and easy with the advance warranty replacements.

Data integrity is a lot more than a technology. It's a philosophy. Only preparation and common sense are essential - everything else can be faked or substituted. My first computer was an Atari 600XL in 1984, and _TO THIS DAY_ I have never had a virus or lost any data. Actually, that's not strictly true - I had a CF card fail in a camera once and lost a few dozen photos (out of several hundred on the card) during the recovery attempt.

Despite have designed, built and used RAID professionally in the past, I don't use it myself and I have about 3TB of photos to archive, and [many] TB of more transient data. All the photos are backed up to a single disk NAS and to Backblaze, and everything is archived every 2-3 months onto a set of HDDs that go into a waterproof container which is then stored in a fireproof safe. (It's not actually a data safe but it's likely good enough). RAID is great in business and enterprise for continuity of data availability... But there are other ways to help with that now.

Oh...
Bonus story

The local IT guys great idea.

I had flown from Auckland to Wellington to do a hardware refresh of a significant sized database and ecommerce system and this enterprise did things very well... They had offices in Akl and Wlg with replicated databases and either centre could operate the business if the other building fell over completely.

However, when I got to the Wellington office at about 5pm on Friday and they had just switched over to the Auckland site answering all the calls and doing all the processing), I noticed a flashing light on their main RAID array so I asked the local IT guy... "Oh, that array kept failing whatever disk was in slot 3 and beeping... Whenever I replaced that disk it never even completed the rebuild and failed so I disabled the beeper"... "When did that happen?"... "Last... June, July, perhaps?"

Sharp intake of breath through teeth. Check backups. Hmm, that's odd. Tape backups haven't worked for months. Local IT person however knew exactly what had happened there. "The tape drive was way too slow, so I changed the backup destination to be drive [whatever, a high letter, like K or something], it has heaps of spare space and is much faster"... You guessed it, yes, that logical drive was on the same physical array that had been degraded for months. While their really critical stuff was replicated to Auckland, all their emails, local documents and heaps of quite important stuff had been spinning around at 15000 RPM without a safety net for months...

"So, you've got a service contract which covers all of this, why didn't you call us?"... "Didn't want to bother you"

Anyway, the next 12 odd hours - from about 7pm until 7am - were spent fixing the tape drive software and creating and testing a full backup, copying all the simple files to a separate machine and then FINALLY, at about 8am the next day, starting the hardware upgrade. It all worked out fine and there was never an interruption to service or data loss, but soooo close.

Cheers - N


14513 posts

Uber Geek
+1 received by user: 3616

Trusted
Subscriber

  Reply # 1434259 25-Nov-2015 01:04
Send private message

USB Drives are not a reliable backup method. In my experience they fail about as much as a floppy disk :) 

Better than nothing, but only JUST.

20193 posts

Uber Geek
+1 received by user: 3787

Trusted
Subscriber

  Reply # 1434271 25-Nov-2015 05:41
Send private message

USB drives fail because people move them and stuff.

I've given up on windows raid 5 now. Too often a drive will get kicked out and force a resync when you reactivate it.

Drives are fine. Just windows being a dick. Wd red so should be fine as far as timeouts etc. Now I just have them all as mirrors. That way I can pull one drive to take off site as a backup and slap a new one in to rebuild.




Richard rich.ms

 1 | 2
View this topic in a long page with up to 500 replies per page Create new topic



Twitter »

Follow us to receive Twitter updates when new discussions are posted in our forums:



Follow us to receive Twitter updates when news items and blogs are posted in our frontpage:



Follow us to receive Twitter updates when tech item prices are listed in our price comparison site:





News »

Intel unveils the 8th Gen Intel Core Processor family for desktop
Posted 25-Sep-2017 19:45


Chow brothers plan to invest NZ$100 million in technology
Posted 24-Sep-2017 16:24


Symantec protects data everywhere with Information Centric Security
Posted 21-Sep-2017 15:33


FUJIFILM introduces X-E3 mirrorless camera with wireless connectivity
Posted 18-Sep-2017 13:53


Vodafone announces new plans with bigger data bundles
Posted 15-Sep-2017 10:51


Skinny launches phone with support for te reo Maori
Posted 14-Sep-2017 08:39


If Vodafone dropping mail worries you, you’re doing online wrong
Posted 11-Sep-2017 13:54


Vodafone New Zealand deploy live 400 gigabit system
Posted 11-Sep-2017 11:07


OPPO camera phones now available at PB Tech
Posted 11-Sep-2017 09:56


Norton Wi-Fi Privacy — Easy, flawed VPN
Posted 11-Sep-2017 09:48


Lenovo reveals new ThinkPad A Series
Posted 8-Sep-2017 14:37


Huawei passes Apple for the first time to capture the second spot globally
Posted 8-Sep-2017 10:45


Vodafone initiative enhances te reo Maori pronunciation on Google Maps
Posted 8-Sep-2017 10:40


Voyager Internet expand local internet phone services company with Conversant acquisition
Posted 6-Sep-2017 18:27


NOW Expands in to Tauranga
Posted 5-Sep-2017 18:16



Geekzone Live »

Try automatic live updates from Geekzone directly in your browser, without refreshing the page, with Geekzone Live now.



Are you subscribed to our RSS feed? You can download the latest headlines and summaries from our stories directly to your computer or smartphone by using a feed reader.

Alternatively, you can receive a daily email with Geekzone updates.