New Geekzone server drives

Forums › Geekzone › New Geekzone server drives

freitasm

BDFL - Memuneh
79000 posts

Uber Geek

Administrator

ID Verified

Trusted

Geekzone

Lifetime subscriber

#279617 28-Oct-2020 16:40

Hi folks

Just a heads up - we have a couple of extra drives being plugged tomorrow (29 Oct) around 10am. Those are hot-plug drives so we shouldn't have a downtime but if we need to restart or anything we might have a short period when the site will stop responding.

I hope a restart will not be needed. The drives are just for backup.

My technology disclosure

Dynamic

3816 posts

Uber Geek

ID Verified

Trusted

Lifetime subscriber

#2592817 28-Oct-2020 16:49

I have no doubt you aren't making my recent error, where I didn't correctly identify a failing server drive. Trying to replace a SAS HDD with a SATA SSD doesn't work!

PSA: On HPE RAID controllers (and I would guess many others as well), you can have a SAS drive array and a SATA drive array working at the same time on the same controller. You can't mix drive types in a single array. I would have sent the SATA drives back, but the helpful client opened them all in advance of my arrival. I should know better than to think 'that job will only take 5 minutes'...

“Don't believe anything you read on the net. Except this. Well, including this, I suppose.” Douglas Adams

Referral links to services I use, really like, and may be rewarded if you sign up:
PocketSmith for budgeting and personal finance management. A great Kiwi company.

GoodSync

GoodSync. Easily back up and sync your files with GoodSync. Simple and secure file backup and synchronisation software will ensure that your files are never lost (affiliate link).

freitasm

BDFL - Memuneh
79000 posts

Uber Geek

Administrator

ID Verified

Trusted

Geekzone

Lifetime subscriber

#2592823 28-Oct-2020 17:08

I am adding larger SAS drives and creating a different logical device for temp backup use. The existing SAS drives from the current backup logical device will be used as hot spare for the main storage logical drive (RAID 5). All drives are SAS and all are HPE originals.

My technology disclosure

nztim

3694 posts

Uber Geek

ID Verified

Trusted

TEAMnetwork

Subscriber

#2592836 28-Oct-2020 17:35

freitasm:
I am adding larger SAS drives and creating a different logical device for temp backup use. The existing SAS drives from the current backup logical device will be used as hot spare for the main storage logical drive (RAID 5). All drives are SAS and all are HPE originals.

Glad to see you look after your servers!

Any views expressed on these forums are my own and don't necessarily reflect those of my employer.

freitasm

BDFL - Memuneh
79000 posts

Uber Geek

Administrator

ID Verified

Trusted

Geekzone

Lifetime subscriber

#2592839 28-Oct-2020 17:39

The main reason to use the existing drives as hot backup and not as storage capacity in the RAID is that all these drives are from the same batch / age - the likelihood of more than one drive failing at the same time is higher than using drives from different batches / ages. This is one of the reasons why so many RAID 5 and RAID 6 fail miserably. This also means I have to order a couple of extra drives now and have them sent to Auckland for use at short notice.

My technology disclosure

nztim

3694 posts

Uber Geek

ID Verified

Trusted

TEAMnetwork

Subscriber

#2592865 28-Oct-2020 18:41

freitasm:
The main reason to use the existing drives as hot backup and not as storage capacity in the RAID is that all these drives are from the same batch / age - the likelihood of more than one drive failing at the same time is higher than using drives from different batches / ages. This is one of the reasons why so many RAID 5 and RAID 6 fail miserably. This also means I have to order a couple of extra drives now and have them sent to Auckland for use at short notice.

The other reason drives fail is smart array and drive firmware not been kept current espionage when HPE release advisories

Any views expressed on these forums are my own and don't necessarily reflect those of my employer.

freitasm

BDFL - Memuneh
79000 posts

Uber Geek

Administrator

ID Verified

Trusted

Geekzone

Lifetime subscriber

#2592872 28-Oct-2020 18:47

I am pretty sure the firmware is updated. Even applied a new firmware to the array last week.

I am really more worried about having all drives being of the same age.

My technology disclosure

PolicyGuy

1706 posts

Uber Geek

ID Verified

Lifetime subscriber

#2592941 28-Oct-2020 20:55

Google published a paper on this quite a few years ago, making this exact point.

I can't find it now, but the two major take-home points were

If a drive fails, on the next day shift, remove all the drives from the same batch and bin them - they will fail soon.
[Wait till the next day shift coz the night shift may not be concentrating properly (it's night time, OK) and might replace the wrong drive]
Never put two drives from the same batch into the same RAID string

Since Google's drive population is statistically indistinguishable from the universe of disk drives, this is soundly based advice

freitasm

BDFL - Memuneh
79000 posts

Uber Geek

Administrator

ID Verified

Trusted

Geekzone

Lifetime subscriber

#2593173 29-Oct-2020 11:49

Thanks all - new drives installed without servers rebooting.

My technology disclosure

gehenna

8434 posts

Uber Geek

Moderator

Trusted

Lifetime subscriber

#2593238 29-Oct-2020 12:39

This is for the subscriber only Plex library content right?

hio77

'That VDSL Cat'
12999 posts

Uber Geek

ID Verified

Trusted

Lizard Networks

Subscriber

#2593362 29-Oct-2020 15:14

more disk space? woo! i can start uploading larger pictures again!

Don't worry @michaelmurfy i'll ensure there are some for you to resize ;)

#include <std_disclaimer>

Any comments made are personal opinion and do not reflect directly on the position my current or past employers may have.

networkn

Networkn
32184 posts

Uber Geek

ID Verified

Trusted

Lifetime subscriber

#2593528 29-Oct-2020 19:23

gehenna:

This is for the subscriber only Plex library content right?

Oh, I thought it was to accomodate the gz pr0n server (which I guess could be served via plex) :)

freitasm

BDFL - Memuneh
79000 posts

Uber Geek

Administrator

ID Verified

Trusted

Geekzone

Lifetime subscriber

#2593699 29-Oct-2020 22:24

Not only had the new drives installed this morning, I've done some maintenance on the existing RAID array (setting a different strip size than previously set) and changing the Read/Write cache ratio (something I should've done a while ago).

And no downtime. Pretty happy with today's work.

My technology disclosure