iServe Cloud Servers down?

Forums › 2degrees (including Slingshot, Orcon, Flip, Stuff Fibre, MyRepublic, 2talk and Vocus) › iServe Cloud Servers down?

HiJinx

253 posts

Ultimate Geek

Geekstore

#54440 17-Dec-2009 20:27

Could just be me, but my cloud server is down, and so is the control panel at https://console.iservices.net.nz/

Also found out that iServe have no after-hours support for incidents like this.. Wonderful.

So, just me, or is it everyone?

Shaun Fisher - www.geekstore.co.nz

E: shaun[at]geekstore[dot]co[dot]nz
P: 0800894508 F: 0800897451

1 | 2

1202 posts

Uber Geek

Trusted

#283530 17-Dec-2009 20:42

Down for me as well. We were only trialing their service but have my doubts now, I can understand a bit of downtime but no after hours support? Do they wait until 8AM to resolve? Hmmmmmmmmmm.

I wonder how long they will be offline for.

Tyler - Parnell Geek - iPhone 3G - Lenovo X301 - Kaseya - Great Western Steak House, these are some of my favourite things.

Sharesies

Trade NZ and US shares and funds with Sharesies (affiliate link).

HiJinx

253 posts

Ultimate Geek

Geekstore

#283534 17-Dec-2009 20:51

I find the fact that there's no after-hours support for not only cloud computing, but anything within their network to be quite a let down to be honest.

Shaun Fisher - www.geekstore.co.nz

E: shaun[at]geekstore[dot]co[dot]nz
P: 0800894508 F: 0800897451

Kraven

708 posts

Ultimate Geek

#283543 17-Dec-2009 21:35

From http://www.iserve.co.nz/announcements.php

Thursday 17 December 20:18
Due to current issues with our cloud and virtual server environment we have disabled administrative access to all cloud and virtual machines.

Technicians are currently investigating and hope to have a resolution within the next 24 hours.

nate

6469 posts

Uber Geek

Moderator

Trusted

Lifetime subscriber

#283545 17-Dec-2009 21:47

Yeah I can't access the control panel either, good thing is our servers are still up.

exportgoldman

1202 posts

Uber Geek

Trusted

#283548 17-Dec-2009 21:51

Kraven: From http://www.iserve.co.nz/announcements.php

Thursday 17 December 20:18
Due to current issues with our cloud and virtual server environment we have disabled administrative access to all cloud and virtual machines.

Technicians are currently investigating and hope to have a resolution within the next 24 hours.

So it looks like they DO have 24 hour monitoring, but 24 hours! Lucky the cloud is still up.

New Zealand's first shaky steps into the cloud.

Tyler - Parnell Geek - iPhone 3G - Lenovo X301 - Kaseya - Great Western Steak House, these are some of my favourite things.

HiJinx

253 posts

Ultimate Geek

Geekstore

#283553 17-Dec-2009 22:07

Kraven: From http://www.iserve.co.nz/announcements.php

Thursday 17 December 20:18
Due to current issues with our cloud and virtual server environment we have disabled administrative access to all cloud and virtual machines.

Technicians are currently investigating and hope to have a resolution within the next 24 hours.

Seems that they posted that at the same time I phoned up Orcon.. Hrm....

Shaun Fisher - www.geekstore.co.nz

E: shaun[at]geekstore[dot]co[dot]nz
P: 0800894508 F: 0800897451

platinumit

1 post

Wannabe Geek

#283856 18-Dec-2009 18:20

Well its now been 24 hours ...My clould server is still floating down river an Iserve have yet to explain why other than technical issues and some bizzare reference to the fact that the admin interface has been disabled and the fact they have had to deploy new hardware

"Our technicians continue to work on a resolution to the issues we have with access to our cloud and VM platforms and are now deploying new hardware to assist in a resolution. We regret the length of time that access to these servers is taking, and deeply apologise for the inconvenience that this is causing our customers"

Phone calls to the helpdesk go along the lines of :

What is wrong ?- We dont know we are not technical
When will it be available - We dont know
Can I speak with someone about it - There is no one available
Can I speak to a manager about it - There are none available they will call you back (they dont)

I have also had -

"All the technical team have left you have called after hours" at 10 mins to closing

"We tried to call you but your number does not work" - All my customers complaining dont seem to have any issue calling me on it

"If you emailed us we would have replied" ...I did twice and they didnt

Your email took 15 mins to get to us - Perhaps because I used your servers

Orcon if your listening - What a lemon you have brought !!

nate

6469 posts

Uber Geek

Moderator

Trusted

Lifetime subscriber

#283870 18-Dec-2009 19:09

platinumit: Well its now been 24 hours ...My clould server is still floating down river an Iserve have yet to explain why other than technical issues and some bizzare reference to the fact that the admin interface has been disabled and the fact they have had to deploy new hardware

I saw your comments that came through on my blog post, thanks for that.

Our servers are still up, console is still down, very concerning considering this is coming up 48 hours.

What's going on iServe?

iserve

8 posts

Wannabe Geek

Trusted

iSERVE Ltd

#283981 19-Dec-2009 11:31

Update on iSERVE Cloud and VM platform access.

On Thursday 17 December, while undertaking routine administration, the storage hosting the cloud virtual machines was found to have a corrupt partition table.

Steps were undertaken after to remedy the problem. At this time access to administer VM and Cloud servers was prohibited to ensure that no service was restarted or shut down, which may have resulted in unrecoverable data loss. Servers remained up and stable during this time.

On the 18th of December, we commenced deploying our secondary SAN as the primary SAN for this platform, and performed a backup of the physical drives from the original san to ensure we could restore files in a worst case scenario . This process did not complete until 3.30am.

At 4am 19th December, We powered off all machines in order to enable us to commence a disk to device backup to the new SAN. It was estimated that the total down time would be around 4.5 hours, however this process is taking longer than expected. We now expect this process to be finished at approximately 4pm this afternoon, at which time we will restore the partition table and metadata backups to restore the partitions and logical volumes. We will advise further once the data has been restored to the new storage device.

exportgoldman

1202 posts

Uber Geek

Trusted

#283986 19-Dec-2009 11:51

iserve: Update on iSERVE Cloud and VM platform access.

On Thursday 17 December, while undertaking routine administration, the storage hosting the cloud virtual machines was found to have a corrupt partition table.

Steps were undertaken after to remedy the problem. At this time access to administer VM and Cloud servers was prohibited to ensure that no service was restarted or shut down, which may have resulted in unrecoverable data loss. Servers remained up and stable during this time....

First off thankyou for the update and to the geeks working hard on this. It seems to be you have it in hand and know what you are doing, the corruption happening in the first place on the SAN is a worry but these things happen I suppose.

The real worry and the thing which is majorly freaking people out is your esculation paths for the cloud seem be shuffling through a normal ISP helpdesk which no one has a clue, so even if you have all this going on everyone thinks no one is working on it and have all gone home. Perhaps time to get a dedicated helpdesk, or keep the current one up with the play.

Thanks again for posting on geekzone, and good luck, your service looks good and I have been trialing it. The SAN corruption does worry us through.

Tyler - Parnell Geek - iPhone 3G - Lenovo X301 - Kaseya - Great Western Steak House, these are some of my favourite things.

iserve

8 posts

Wannabe Geek

Trusted

iSERVE Ltd

#283993 19-Dec-2009 12:18

exportgoldman:
The real worry and the thing which is majorly freaking people out is your esculation paths for the cloud seem be shuffling through a normal ISP helpdesk which no one has a clue, so even if you have all this going on everyone thinks no one is working on it and have all gone home. Perhaps time to get a dedicated helpdesk, or keep the current one up with the play.

Thanks again for posting on geekzone, and good luck, your service looks good and I have been trialing it. The SAN corruption does worry us through.

Support for the prouduct is via our normal support channels published here:http://www.iserve.co.nz/contactus.php

We do have a dedicated helpdesk team available via the iserve contact details and I am aware that platinum have experienced an issue due to high call volumes. When the call volume is high, we overflow our inbound calls to an external call centre. The operator that Platinum spoke to advised him incorrectly of our support levels, staff are on site untill 7pm, and we monitor the services 24/7 - yes techs were here and working on it, unfortunately ( lets call it murphys law) Help Desk numbers were down due to illness - and we are loathed to put senior technicians on the phones to answer customers questions, when their skills are required to resolve the issue. Obviously the time spent on the phone, is time not spent on resolving the issue.

On a general note, the iSERVE hosting support team has been light on resource for quite some time, however that has been addressed, and an improvement will be noticeable shortly.

iSERVE's support team were up to date with the progress, and Orcons support team were informed of the issues, however they would not have been able to offer any technical update or assistance with the problem - we didn't anticipate customers to be directing their inquiries to Orcon, and is something we will put consideration too for the future.

With regards to the data corruption, we believe that it is a freak incident, and we are all very lucky that the systems administrators we have were diligent enough to discover it before the issue became more serious. We are of course in communication with Citrix, and will continue to discuss it with them.

I hope the above is legible - like all our techs, there has been very little sleep had over the past 2 nights

nate

6469 posts

Uber Geek

Moderator

Trusted

Lifetime subscriber

#283999 19-Dec-2009 13:23

Just spoke to the iServe ops manager, hopefully servers will be back up at 4pm.

/fingers crossed.

HiJinx

253 posts

Ultimate Geek

Geekstore

#284051 19-Dec-2009 17:27

---update--- 4.59 19 December
The backup process continues, with close to 90% of the process complete, we now estimate it will finish at approx 6.45pm

Source: http://www.iserve.co.nz/announcements.php

Shaun Fisher - www.geekstore.co.nz

E: shaun[at]geekstore[dot]co[dot]nz
P: 0800894508 F: 0800897451

HiJinx

253 posts

Ultimate Geek

Geekstore

#284054 19-Dec-2009 17:39

@iserve

Will we be getting any sort of service credit for this trouble?

Shaun Fisher - www.geekstore.co.nz

E: shaun[at]geekstore[dot]co[dot]nz
P: 0800894508 F: 0800897451

iserve

8 posts

Wannabe Geek

Trusted

iSERVE Ltd

#284055 19-Dec-2009 17:44

re compensation:

we intend to compensate customers appropriately, (even though our Ts & Cs don't require us to), and we'll make that decision once everything is restored

1 | 2