Big NZ Government website outage

Forums › IT Pro and developers › Big NZ Government website outage

1 | 2

jonb

1796 posts

Uber Geek
+1 received by user: 545

Trusted

#526200 27-Sep-2011 13:59

Geekzone's Juha has written an article about it for The Register:

http://www.theregister.co.uk/2011/09/26/kiwi_gov_stuffs_upgrade/

freitasm

BDFL - Memuneh
80652 posts

Uber Geek
+1 received by user: 41045

Administrator

ID Verified

Trusted

Geekzone

Lifetime subscriber

#526273 27-Sep-2011 15:54

And then I got this press release today:

National Cyber Security Centre opens

The new National Cyber Security Centre (NCSC) has officially opened today.

The centre is charged with helping to defend government agencies and critical infrastructure providers against cyber threats.

The Minister for Communications and Information Technology, Steven Joyce, says the global threat from cyber intrusions is real and growing, and New Zealanders and the New Zealand economy are not immune.

“Cyber security is becoming increasingly important for New Zealanders, businesses and government. Cyber intrusions have the potential to impact on the reliability of critical infrastructure, government services, and the economy.”

www.ncsc.govt.nz

I wonder if they have planned for internal security problems too...

Referral links: Quic Broadband (free setup code: R587125ERQ6VE) | Samsung | AliExpress | Wise | Sharesies

Support Geekzone by subscribing (browse ads-free), or making a one-off or recurring donation through PressPatron.

magu

Professional yak shaver
1599 posts

Uber Geek
+1 received by user: 7

Trusted

BitSignal

Lifetime subscriber

#526275 27-Sep-2011 15:56

Someone was mentioning on Twitter that it was a SAN-related issue (attempted upgrades failed). Can anyone verify here?

"Roads? Where we're going, we don't need roads." - Doc Emmet Brown

Ragnor

8279 posts

Uber Geek
+1 received by user: 585

Trusted

#526296 27-Sep-2011 16:31

Sounds plausible because they would presumably not have any other single points of failure (webservers, db servers etc).

I remember we got a quote for a SAN a little while ago, had meeting with the vendor/reseller and it occurred to me that a SAN is a single point of failure so I asked specifically what happens if the SAN fails and they were confident that SAN's don't fail (redundant... psu's, boards, disks) I was like like O'RLY!!

Two SAN's weren't in out budget and and even one SAN was probably overkill pricewise for this project, we went with a set of NAS units instead.

Always have two of everything minimum imo!

magu

Professional yak shaver
1599 posts

Uber Geek
+1 received by user: 7

Trusted

BitSignal

Lifetime subscriber

#526297 27-Sep-2011 16:32

Everything fails. You just need enough units to be able to handle the outage. If they did have a SAN outage, they clearly did not have enough units.

I know the feeling of a botched SAN upgrade, though.

"Roads? Where we're going, we don't need roads." - Doc Emmet Brown

chchrob

23 posts

Geek

Trusted

#527314 29-Sep-2011 16:52

I have been scratching my head as to what would cause such an extended outage on what you would have to assume is a redundant platform with decent hardware and systems behind it.

I am not an Engineer, but if I had to have a guess...I would say SAN failure or large scale data-loss and it has taken this long to restore and verify data. Large scale restoration can be a slow and painful process. Although a decent SAN should make this pretty damn unlikely!

All things aside, gotta feel sorry for whoever is looking after this it wouldn't be fun.

COO @ Digiweb Group / Web Drive

Geekzone

Want to support Geekzone and browse the site without the ads? Subscribe to Geekzone now (monthly, annual and lifetime options).

insane

3324 posts

Uber Geek
+1 received by user: 1006

ID Verified

Trusted

2degrees

Subscriber

#528147 2-Oct-2011 01:51

If it was a SAN upgrade that took them out then there are so many things which could have gone wrong, the list is long. Any storage upgrades are tricky and all consist of a 'hold your breath' moment, specially at the point when you're running on a single active controller during the upgrades in the case of firmware updates.

The sad truth is that not many of these government projects have true redundancy as often each part of the solution is built using hardware from different vendors which does not allow for storage replication and auto site fail over etc.

I can't think of many providers of hosted solutions who don't have their own storage related story/nightmare to tell.

1 | 2