Imagine what the meeting will be like this morning with AirNZ and IBM

Forums › IT Pro and developers › Imagine what the meeting will be like this morning with AirNZ and IBM

CB_24

371 posts

Ultimate Geek
+1 received by user: 33

#42825 12-Oct-2009 11:03

Would love to be sitting in on that one!

[Mod (N8): Moved forums]

1 | 2

3352 posts

Uber Geek
+1 received by user: 644

ID Verified

Trusted

Lifetime subscriber

#263467 12-Oct-2009 12:27

Yep. I would love to be there too. It was pretty chaotic yesterday. Very unfortunate day to happen on being the last day of the holidays.

My views (except when I am looking out their windows) are not those of my employer.

old3eyes

9158 posts

Uber Geek
+1 received by user: 1364

Subscriber

#263476 12-Oct-2009 12:55

I guess AIR NZ problems weren't made any better due to the fact that they have gotten rid of most manual checkin desks. Automated checkin only these days..

Regards,

Old3eyes

corksta

2405 posts

Uber Geek
+1 received by user: 382

Trusted

Subscriber

#263477 12-Oct-2009 12:56

If the tone of this email is carried into the meeting it would be an absolute ripper!

http://www.stuff.co.nz/business/industries/2955289/Air-New-Zealand-boss-criticises-IBM-over-outage

(sorry, javascript disabled on work computer so can't put it in as a proper link)

JasonDarwin

33 posts

Geek

#263479 12-Oct-2009 13:01

doubt we'll get to know the true stary

drspangle

67 posts

Master Geek
+1 received by user: 2

ID Verified

#263482 12-Oct-2009 13:18

We aren't all running around in a blind panic...yet

Se7ensyns

133 posts

Master Geek

#263484 12-Oct-2009 13:24

Well I can say that AIr NZ were not the only company impacted as the IBM data center holds many differnet companyies systems. Ours were impacted but thank god they had redundancy and the majority were not customer facing.

The email tells you what happened. There is a little more to it but essentialy in laymans terms what was said is correct.

I know I was very busy yesterday arvo trying to sort out engineers and parts replacements due to blown power supplies etc.

New World

Shop on-line at New World now for your groceries (affiliate link).

patatrat

261 posts

Ultimate Geek

#263485 12-Oct-2009 13:24

I imagine it will go like this:
AirNZ: Why haven't you provided DR?
IBM: You haven't paid for it.
AirNZ: Ah, right.

Se7ensyns

133 posts

Master Geek

#263489 12-Oct-2009 13:50

LOL you are probably right there

nate

6473 posts

Uber Geek
+1 received by user: 458

Retired Mod

Trusted

Lifetime subscriber

#263512 12-Oct-2009 15:54

corksta: If the tone of this email is carried into the meeting it would be an absolute ripper!

http://www.stuff.co.nz/business/industries/2955289/Air-New-Zealand-boss-criticises-IBM-over-outage

There are some absolute gems in that news article about IBM including:

I am struggling to recall a time where I have seen a supplier so slow to react to a catastrophic system failure such as this and so unwilling to accept responsibility and apologise to its client and its client's customers

and

I have been left with no option but to ask the IT team to review the full range of options available to us

Oops and ouch.

chiefie

I iz your trusted friend
5877 posts

Uber Geek
+1 received by user: 151

Retired Mod

Trusted

Lifetime subscriber

#263520 12-Oct-2009 16:19

We had IBM SANs and that when horribly wrong on two Fridays ago... and the crew still trying to fix it... what happened was the SANs' dual controllers gone dead at the same time, chances of that happening is really really really really rare!

Internet is my backyard...

«Geekzone blog: Tech 'n Chips Takeaway» «Personal blog: And then...»

Please read the Geekzone's FUG

corksta

2405 posts

Uber Geek
+1 received by user: 382

Trusted

Subscriber

#263524 12-Oct-2009 16:46

nate:
corksta: If the tone of this email is carried into the meeting it would be an absolute ripper!

http://www.stuff.co.nz/business/industries/2955289/Air-New-Zealand-boss-criticises-IBM-over-outage

There are some absolute gems in that news article about IBM including:

I am struggling to recall a time where I have seen a supplier so slow to react to a catastrophic system failure such as this and so unwilling to accept responsibility and apologise to its client and its client's customers

and

I have been left with no option but to ask the IT team to review the full range of options available to us

Oops and ouch.

Agreed! My fiance is a flight attendant for Air NZ and got home at 3.30am this morning (delayed 4 hours) so she was having a chuckle too. His internal emails are normally quite placid so this must have really riled him up!

Dell

Shop now for Dell laptops and other devices (affiliate link).

wellygary

8810 posts

Uber Geek
+1 received by user: 5287

#263526 12-Oct-2009 17:06

You sorta wonder if the intitial call from Air NZ to IBM was picked up by some low level desk warmer, who on a Sunday afternoon replied with something like, "its the weekend call back monday" :D

freitasm

BDFL - Memuneh
80646 posts

Uber Geek
+1 received by user: 41030

Administrator

ID Verified

Trusted

Geekzone

Lifetime subscriber

#263527 12-Oct-2009 17:12

wellygary: You sorta wonder if the intitial call from Air NZ to IBM was picked up by some low level desk warmer, who on a Sunday afternoon replied with something like, "its the weekend call back monday" :D

Or "Have you tried turning the computer off and on again?"

Referral links: Quic Broadband (free setup code: R587125ERQ6VE) | Samsung | AliExpress | Wise | Sharesies

Support Geekzone by subscribing (browse ads-free), or making a one-off or recurring donation through PressPatron.

Mark

1653 posts

Uber Geek
+1 received by user: 555

#263720 13-Oct-2009 19:01

I'm quite surprised it all went so pear shaped, when I was on the AirNZ account at IBM it was all reasonably well setup and they'd had power outages before that they'd survived .... guess a lot changes in 5 years though.

You can bet the "no apologies from IBM" part is because the IBM lawyers would forbid anyone apologising because it admits blame, there will be the root cause analysis and follow on investigations to do before any says anything.

Mark

insane

3324 posts

Uber Geek
+1 received by user: 1006

ID Verified

Trusted

2degrees

Subscriber

#263927 14-Oct-2009 18:17

nate:
corksta: If the tone of this email is carried into the meeting it would be an absolute ripper!

http://www.stuff.co.nz/business/industries/2955289/Air-New-Zealand-boss-criticises-IBM-over-outage

There are some absolute gems in that news article about IBM including:

I am struggling to recall a time where I have seen a supplier so slow to react to a catastrophic system failure such as this and so unwilling to accept responsibility and apologise to its client and its client's customers

Oops and ouch.

Well interestingly enough I was having a discussion on sunday morning about VMware and virtualisation etc with one of the IBM on call engineers when he got a very frantic phone call advising him of multiple systems down. He junmped up got his PC/Mac from this car and plugged in the Vodafone wireless connection and started looking at all the virtual machines which were down, after which he made anther call and had to go to the data centre.

Hardly slow response from the Engineer. I suspect it took a while for IBM to contact their techs to sort the problem out from what it sounds.

Interestingly enough that data centre is being decommissioned from what I hear and all services pulled over to the new Orbit Data Centre in Albany which is Tier 3+ versus only Tier 2.

Without knowing what SLA was sold to Air New Zealand and what they paid for in terms of DR, no ammount of finger pointing will fix.

1 | 2