Five words to screw your day: "Enact your disaster recovery plans"

Forums › IT Pro and developers › Five words to screw your day: "Enact your disaster recovery plans"

1 | 2

timmmay

20859 posts

Uber Geek
+1 received by user: 5350

Trusted

Lifetime subscriber

#2671470 11-Mar-2021 09:59

wellygary:

So... Does this indicate that the claims of "the cloud has redundancy" only extend so far and any redundancy might still be on the same physical site??...

Betcha that multi-site cloud replication has just become a much more desirable service tier in Europe...( and globally)

All major cloud providers recommend geographic redundancy. AWS does it with availability zones, which are data centers within a region connected by high speed low latency fiber but are on different flood planes - in practice they're 1 - 50km apart. Azure and Google probably similar concepts. Anyone hosting in a single data center takes a risk. Backups are also still required for services in the cloud.

JaseNZ

2576 posts

Uber Geek
+1 received by user: 1489

ID Verified

Lifetime subscriber

#2671521 11-Mar-2021 10:09

sbiddle:

Halon has been banned in most parts of the world for probably 25 years now for new builds. There simply aren't alternatives that are as good as a halon gas flood system.

If the fire was a large scale electrical fire (as some have talked about) we don't even know if they would have been effective anyway.

I did not know that I thought it was still used.

Ding Ding Ding Ding Ding : Ice cream man , Ice cream man

JaseNZ

2576 posts

Uber Geek
+1 received by user: 1489

ID Verified

Lifetime subscriber

#2671522 11-Mar-2021 10:12

As big as they were would they have redundant off site backup ??

Ding Ding Ding Ding Ding : Ice cream man , Ice cream man

timmmay

20859 posts

Uber Geek
+1 received by user: 5350

Trusted

Lifetime subscriber

#2671545 11-Mar-2021 10:39

JaseNZ:

As big as they were would they have redundant off site backup ??

Many cloud providers make it fairly simple for you to mirror / backup your data to other data centers but don't do it automatically.

floydbloke

3646 posts

Uber Geek
+1 received by user: 4554

ID Verified

#2671600 11-Mar-2021 11:11

timmmay:

JaseNZ:

As big as they were would they have redundant off site backup ??

Many cloud providers make it fairly simple for you to mirror / backup your data to other data centers but don't do it automatically.

..and there is a cost.

Sometimes I use big words I don't always fully understand in an effort to make myself sound more photosynthesis.

Beccara

1473 posts

Uber Geek
+1 received by user: 517

ID Verified

#2671604 11-Mar-2021 11:24

OVH is low cost cloud provider, it's upto the end user to design their application in this case around geo-redundancy requirements. They are not the solution for you if you dont want to do the heavy lifting around design and expect them to, Hell even Azure/AWS still require a decent amount of thought to DRP/B&R

Most problems are the result of previous solutions...

All comment's I make are my own personal opinion and do not in any way, shape or form reflect the views of current or former employers unless specifically stated

Apple TV

Stream your favourite shows now on Apple TV (affiliate link).

Beccara

1473 posts

Uber Geek
+1 received by user: 517

ID Verified

#2671608 11-Mar-2021 11:28

I think it's also worth pointing out that gas suppression systems depending on how they are designed aren't expecting fire to restart after the dump and also only have a fixed volume of gas to deploy so any leaks in the building may impact cover. They also dont help if the fire brings it's own oxidizer to the party like chemicals and battery fires

freitasm

BDFL - Memuneh
80656 posts

Uber Geek
+1 received by user: 41057

Administrator

ID Verified

Trusted

Geekzone

Lifetime subscriber

#2671615 11-Mar-2021 11:43

The datacentre fire was mighty impressive (a few more photos).

Referral links: Quic Broadband (free setup code: R587125ERQ6VE) | Samsung | AliExpress | Wise | Sharesies

Support Geekzone by subscribing (browse ads-free), or making a one-off or recurring donation through PressPatron.

hio77

'That VDSL Cat'
13036 posts

Uber Geek
+1 received by user: 3896

ID Verified

Trusted

Lizard Networks

Subscriber

#2671677 11-Mar-2021 13:33

As someone who works in this space, it's honestly pretty impressive how fast they are moving to restore what they can.

It's a chilling reminder that even a datacenter, isn't immune to disasters.

Twitter is a bit of a laugh to read, you do see the folk who bought up big in one DC and are now feeling the pain... their NOC and support folk must be sick of answering customers who think data is 100% safe in a DC.

#include <std_disclaimer>

Any comments made are personal opinion and do not reflect directly on the position my current or past employers may have.

michaelmurfy

meow
13581 posts

Uber Geek
+1 received by user: 10914

Moderator

ID Verified

Trusted

Lifetime subscriber

#2671703 11-Mar-2021 15:17

hio77:

It's a chilling reminder that even a datacenter, isn't immune to disasters.

As somebody who worked in a datacentre in Christchurch during the Christchurch quakes I remember going through the datacentre after the quake to asses any damage and saw a whole bunch of drives with drive shock (failed status). I ended up going through the datacentre over a week replacing drives for customers.

I was not lucky - my data was on 2 drives (2 different locations) which were spun up at the time of the quake. I lost all my family photos so basically I have no photos before the Christchurch quake. Cloud backups, despite me working in a datacentre at the time wasn't a big thing due to the limited upload speed of ADSL.

I seriously can't stress enough to backup offsite outside of the country. I've been there and lost data. I don't want anyone else to as even to this day I miss the memories I lost.

These days my vitally important data is in 3 physical places within NZ as well as encrypted and backed up to Crashplan.

I've found out I had 2 servers in the affected datacentre (that are now offline) so I presume they're gone. I had no data on them that was important and experienced no downtime - replacement servers were spun up in other datacentres a few hours later, assemble scripts run to provision and move data across and they were up serving people very shortly after - in the meantime the other servers I had in other datacentres could handle all the load.

Sometimes you have to go through data loss to plan for it in the future. While I experienced data loss in the past, I didn't experience any yesterday.

Michael Murphy | https://murfy.nz
Referral Links: Quic Broadband (use R122101E7CV7Q for free setup)

Are you happy with what you get from Geekzone? Please consider supporting us by subscribing.
Opinions are my own and not the views of my employer.

dfnt

1553 posts

Uber Geek
+1 received by user: 1036

Trusted

Lifetime subscriber

#2671884 11-Mar-2021 19:57

I have/had a server in SGB1, no idea of its status but probably safe to assume its gone. No important data on it, and have just stood up a replacement in RBX..

Geekzone support

Support Geekzone with one-off or recurring donations Donate via PressPatron.

yitz

2239 posts

Uber Geek
+1 received by user: 594

#2672058 12-Mar-2021 01:52

Thinking of all those lost seedboxes 🤔

...and the offline speedtest server of course 😉

Baboon

386 posts

Ultimate Geek
+1 received by user: 35

#2672060 12-Mar-2021 05:17

I find it disturbing that in 2021 some companies still consider redundancy to be two copies within visual range of each other :-( Or consider 'being in the cloud' safe enough for their data, even if the 'cloud' is actually one physical location.

"The surest sign that intelligent life exists elsewhere in the universe is that it has never tried to contact us."

- Calvin and Hobbes (Bill Watterson)

Beccara

1473 posts

Uber Geek
+1 received by user: 517

ID Verified

#2672193 12-Mar-2021 10:43

Alot of companies still see IT as a budget blackhole and/or have the idea that they functioned perfectly fine before computers so if the computer go down for a week it wont be a problem. Without the will to spend even having 2 copies of the data is often not a thing

1 | 2