iServe Cloud Servers down?

Forums › 2degrees (including Slingshot, Orcon, Flip, Stuff Fibre, MyRepublic, 2talk and Vocus) › iServe Cloud Servers down?

1 | 2

HiJinx

253 posts

Ultimate Geek

Geekstore

#284056 19-Dec-2009 17:49

iserve: re compensation:

we intend to compensate customers appropriately, (even though our Ts & Cs don't require us to), and we'll make that decision once everything is restored

Quick response, much appreciated. =]

Shaun Fisher - www.geekstore.co.nz

E: shaun[at]geekstore[dot]co[dot]nz
P: 0800894508 F: 0800897451

exportgoldman

1202 posts

Uber Geek
+1 received by user: 3

Trusted

#284062 19-Dec-2009 18:58

iserve: We do have a dedicated helpdesk team available via the iserve contact details and I am aware that platinum have experienced an issue due to high call volumes. When the call volume is high, we overflow our inbound calls to an external call centre. The operator that Platinum spoke to advised him incorrectly of our support levels, staff are on site untill 7pm, and we monitor the services 24/7 - yes techs were here and working on it, unfortunately ( lets call it murphys law) Help Desk numbers were down due to illness - and we are loathed to put senior technicians on the phones to answer customers questions, when their skills are required to resolve the issue. Obviously the time spent on the phone, is time not spent on resolving the issue.

I hope the above is legible - like all our techs, there has been very little sleep had over the past 2 nights

I now actually feel more confident entrusting my virtual servers with iServe from your posts and whats happened. Your team seem very good and we will start hosting solutions with you in the New Year.

Hopefully you will be getting some sleep soon :-)

Tyler - Parnell Geek - iPhone 3G - Lenovo X301 - Kaseya - Great Western Steak House, these are some of my favourite things.

HiJinx

253 posts

Ultimate Geek

Geekstore

#284076 19-Dec-2009 21:16

----update----- 08:40 19 December 2009
The downtime has unfortunately exceeded our estimates but will be completed as soon as possible.

Source: http://www.iserve.co.nz/announcements.php

Shaun Fisher - www.geekstore.co.nz

E: shaun[at]geekstore[dot]co[dot]nz
P: 0800894508 F: 0800897451

HiJinx

253 posts

Ultimate Geek

Geekstore

#284081 19-Dec-2009 21:38

---UPDATE--- 9:00 19 December
All nodes are back opperational and service have been restored.
All customer machines that were active when the fault occured have been restarted.
All other customers requiring access to their suspended machines can now ssh / r-desktop into them and resume business as usual.

We assure you that the fault that has occured this time should not occur again in the future.

We apologise to all customers for the inconvenience that this has caused and appreciate your patience during this time.

Source: http://www.iserve.co.nz/announcements.php

Shaun Fisher - www.geekstore.co.nz

E: shaun[at]geekstore[dot]co[dot]nz
P: 0800894508 F: 0800897451

nate

6473 posts

Uber Geek
+1 received by user: 458

Retired Mod

Trusted

Lifetime subscriber

#284186 20-Dec-2009 13:50

iserve: we intend to compensate customers appropriately, (even though our Ts & Cs don't require us to), and we'll make that decision once everything is restored

Good to hear, considering our server went down at 3:54AM, then back online at 9:05PM, some 17 hours later...

I'm just glad it was a weekend, and not during the week.

berend

62 posts

Master Geek
+1 received by user: 1

#284333 21-Dec-2009 11:20

What I don't understand is why a partition failure is such a big deal?

I mean, they have only one disk???

Samsung

Shop now on Samsung phones, tablets, TVs and more (affiliate link).

Simonm

181 posts

Master Geek

#284401 21-Dec-2009 16:41

It will be a partition failure on the centralised storage they use.

When a centralised storage system goes screwy like that, it is generally a massive job to get everything up and running again. Unfortunately the more levels of redundancy in a system, the more chance of the data getting screwed. And the greater risk of human error.

berend

62 posts

Master Geek
+1 received by user: 1

#284403 21-Dec-2009 16:50

Simonm: It will be a partition failure on the centralised storage they use.

Well of course. But it's insane they have only one.

iserve

8 posts

Wannabe Geek

Trusted

iSERVE Ltd

#284409 21-Dec-2009 17:11

berend: What I don't understand is why a partition failure is such a big deal?

I mean, they have only one disk???

Thats not correct, infact there were 2 SANS in play at the time, both of which of course have multiple drives

I have been corrected on my use of the word 'Partition" infact it was the Meta Data which was corrupt - as we have previously mentioned restarting any service attached to this, could result in the mapping of content being lost.

While the total down time for services was extensive, we needed to be ultra cautious in the resolution, this is much more favorable to our clients than knowing about the issue, and not proactively resolving it, meaning certain data loss for some or all clients.

1 | 2