Geekzone: technology news, blogs, forums
Guest
Welcome Guest.
You haven't logged in yet. If you don't have an account you can register now.




253 posts

Ultimate Geek

Geekstore

Topic # 54440 17-Dec-2009 20:27
Send private message

Could just be me, but my cloud server is down, and so is the control panel at https://console.iservices.net.nz/

Also found out that iServe have no after-hours support for incidents like this.. Wonderful.

So, just me, or is it everyone?




Shaun Fisher - www.geekstore.co.nz

E: shaun[at]geekstore[dot]co[dot]nz
P: 0800894508  F: 0800897451

View this topic in a long page with up to 500 replies per page Create new topic
 1 | 2
1200 posts

Uber Geek
+1 received by user: 3

Trusted

  Reply # 283530 17-Dec-2009 20:42
Send private message


Down for me as well. We were only trialing their service but have my doubts now, I can understand a bit of downtime but no after hours support? Do they wait until 8AM to resolve? Hmmmmmmmmmm.

I wonder how long they will be offline for.




Tyler - Parnell Geek - iPhone 3G - Lenovo X301 - Kaseya - Great Western Steak House, these are some of my favourite things.



253 posts

Ultimate Geek

Geekstore

  Reply # 283534 17-Dec-2009 20:51
Send private message

I find the fact that there's no after-hours support for not only cloud computing, but anything within their network to be quite a let down to be honest.




Shaun Fisher - www.geekstore.co.nz

E: shaun[at]geekstore[dot]co[dot]nz
P: 0800894508  F: 0800897451

 
 
 
 


561 posts

Ultimate Geek
+1 received by user: 62


  Reply # 283543 17-Dec-2009 21:35
Send private message

From http://www.iserve.co.nz/announcements.php

Thursday 17 December 20:18
Due to current issues with our cloud and virtual server environment we have disabled administrative access to all cloud and virtual machines.

Technicians are currently investigating and hope to have a resolution within the next 24 hours.

I'm @nate
6290 posts

Uber Geek
+1 received by user: 374

Moderator
Trusted
Subscriber

  Reply # 283545 17-Dec-2009 21:47
Send private message

Yeah I can't access the control panel either, good thing is our servers are still up.




1200 posts

Uber Geek
+1 received by user: 3

Trusted

  Reply # 283548 17-Dec-2009 21:51
Send private message

Kraven: From http://www.iserve.co.nz/announcements.php

Thursday 17 December 20:18
Due to current issues with our cloud and virtual server environment we have disabled administrative access to all cloud and virtual machines.

Technicians are currently investigating and hope to have a resolution within the next 24 hours.


So it looks like they DO have 24 hour monitoring, but 24 hours! Lucky the cloud is still up.

New Zealand's first shaky steps into the cloud.




Tyler - Parnell Geek - iPhone 3G - Lenovo X301 - Kaseya - Great Western Steak House, these are some of my favourite things.



253 posts

Ultimate Geek

Geekstore

  Reply # 283553 17-Dec-2009 22:07
Send private message

Kraven: From http://www.iserve.co.nz/announcements.php

Thursday 17 December 20:18
Due to current issues with our cloud and virtual server environment we have disabled administrative access to all cloud and virtual machines.

Technicians are currently investigating and hope to have a resolution within the next 24 hours.


Seems that they posted that at the same time I phoned up Orcon.. Hrm.... Undecided




Shaun Fisher - www.geekstore.co.nz

E: shaun[at]geekstore[dot]co[dot]nz
P: 0800894508  F: 0800897451

1 post

Wannabe Geek


  Reply # 283856 18-Dec-2009 18:20
Send private message

Well its now been 24 hours ...My clould server is still floating down river an Iserve have yet to explain why other than technical issues and some bizzare reference to the fact that the admin interface has been disabled and the fact they have had to deploy new hardware

"Our technicians continue to work on a resolution to the issues we have with access to our cloud and VM platforms and are now deploying new hardware to assist in a resolution. We regret the length of time that access to these servers is taking, and deeply apologise for the inconvenience that this is causing our customers"



Phone calls to the helpdesk go along the lines of :

What is wrong ?- We dont know we are not technical
When will it be available - We dont know
Can I speak with someone about it - There is no one available
Can I speak to a manager about it - There are none available they will call you back (they dont)

I have also had -

"All the technical team have left you have called after hours" at 10 mins to closing

"We tried to call you but your number does not work" - All my customers complaining dont seem to have any issue calling me on it 

"If you emailed us we would have replied" ...I did twice and they didnt

Your email took 15 mins to get to us - Perhaps because I used your servers  



Orcon if your listening - What a lemon you have brought !!

I'm @nate
6290 posts

Uber Geek
+1 received by user: 374

Moderator
Trusted
Subscriber

  Reply # 283870 18-Dec-2009 19:09
Send private message

platinumit: Well its now been 24 hours ...My clould server is still floating down river an Iserve have yet to explain why other than technical issues and some bizzare reference to the fact that the admin interface has been disabled and the fact they have had to deploy new hardware


I saw your comments that came through on my blog post, thanks for that.

Our servers are still up, console is still down, very concerning considering this is coming up 48 hours.

What's going on iServe?




8 posts

Wannabe Geek

Trusted
iSERVE Ltd

  Reply # 283981 19-Dec-2009 11:31
Send private message

Update on iSERVE Cloud and VM platform access.

On Thursday 17 December, while undertaking routine administration, the storage hosting the cloud virtual machines was found to have a corrupt partition table.

Steps were undertaken after to remedy the problem. At this time access to administer VM and Cloud servers was prohibited to ensure that no service was restarted or shut down, which may have resulted in unrecoverable data loss. Servers remained up and stable during this time.

On the 18th of December, we commenced deploying our secondary SAN as the primary SAN for this platform, and performed a backup of the physical drives from the original san to ensure we could restore files in a worst case scenario . This process did not complete until 3.30am.

At 4am 19th December, We powered off all machines in order to enable us to commence a disk to device backup to the new SAN. It was estimated that the total down time would be around 4.5 hours, however this process is taking longer than expected. We now expect this process to be finished at approximately 4pm this afternoon, at which time we will restore the partition table and metadata backups to restore the partitions and logical volumes. We will advise further once the data has been restored to the new storage device.

1200 posts

Uber Geek
+1 received by user: 3

Trusted

  Reply # 283986 19-Dec-2009 11:51
Send private message

iserve: Update on iSERVE Cloud and VM platform access.

On Thursday 17 December, while undertaking routine administration, the storage hosting the cloud virtual machines was found to have a corrupt partition table.

Steps were undertaken after to remedy the problem. At this time access to administer VM and Cloud servers was prohibited to ensure that no service was restarted or shut down, which may have resulted in unrecoverable data loss. Servers remained up and stable during this time....


First off thankyou for the update and to the geeks working hard on this. It seems to be you have it in hand and know what you are doing, the corruption happening in the first place on the SAN is a worry but these things happen I suppose.

The real worry and the thing which is majorly freaking people out is your esculation paths for the cloud seem be shuffling through a normal ISP helpdesk which no one has a clue, so even if you have all this going on everyone thinks no one is working on it and have all gone home. Perhaps time to get a dedicated helpdesk, or keep the current one up with the play.

Thanks again for posting on geekzone, and good luck, your service looks good and I have been trialing it. The SAN corruption does worry us through.




Tyler - Parnell Geek - iPhone 3G - Lenovo X301 - Kaseya - Great Western Steak House, these are some of my favourite things.

8 posts

Wannabe Geek

Trusted
iSERVE Ltd

  Reply # 283993 19-Dec-2009 12:18
Send private message

exportgoldman:
The real worry and the thing which is majorly freaking people out is your esculation paths for the cloud seem be shuffling through a normal ISP helpdesk which no one has a clue, so even if you have all this going on everyone thinks no one is working on it and have all gone home. Perhaps time to get a dedicated helpdesk, or keep the current one up with the play.

Thanks again for posting on geekzone, and good luck, your service looks good and I have been trialing it. The SAN corruption does worry us through.



Support for the prouduct is via our normal support channels published here:http://www.iserve.co.nz/contactus.php


We do have a dedicated helpdesk team available via the iserve contact details and I am aware that platinum have experienced an issue due to high call volumes.  When the call volume is high, we overflow our inbound calls to an external call centre.  The operator that Platinum spoke to advised him incorrectly of our support levels, staff are on site untill 7pm, and we monitor the services 24/7 - yes techs were here and working on it, unfortunately ( lets call it murphys law) Help Desk numbers were down due to illness - and we are loathed to put senior technicians on the phones to answer customers questions, when their skills are required to resolve the issue.  Obviously the time spent on the phone, is time not spent on resolving the issue.


On a general note, the iSERVE hosting support team has been light on resource for quite some time, however that has been addressed, and an improvement will be noticeable shortly.


iSERVE's support team were up to date with the progress, and Orcons support team were informed of the issues, however they would not have been able to offer any technical update or assistance with the problem - we didn't anticipate customers to be directing their inquiries to Orcon, and is something we will put consideration too for the future.


With regards to the data corruption, we believe that it is a freak incident, and we are all very lucky that the systems administrators we have were diligent enough to discover it before the issue became more serious. We are of course in communication with Citrix, and will continue to discuss it with them.


I hope the above is legible - like all our techs, there has been very little sleep had over the past 2 nights

I'm @nate
6290 posts

Uber Geek
+1 received by user: 374

Moderator
Trusted
Subscriber

  Reply # 283999 19-Dec-2009 13:23
Send private message

Just spoke to the iServe ops manager, hopefully servers will be back up at 4pm.

/fingers crossed.






253 posts

Ultimate Geek

Geekstore

  Reply # 284051 19-Dec-2009 17:27
Send private message

---update--- 4.59 19 December
The backup process continues, with close to 90% of the process complete, we now estimate it will finish at approx 6.45pm

Source: http://www.iserve.co.nz/announcements.php




Shaun Fisher - www.geekstore.co.nz

E: shaun[at]geekstore[dot]co[dot]nz
P: 0800894508  F: 0800897451



253 posts

Ultimate Geek

Geekstore

  Reply # 284054 19-Dec-2009 17:39
Send private message

@iserve

Will we be getting any sort of service credit for this trouble?




Shaun Fisher - www.geekstore.co.nz

E: shaun[at]geekstore[dot]co[dot]nz
P: 0800894508  F: 0800897451

8 posts

Wannabe Geek

Trusted
iSERVE Ltd

  Reply # 284055 19-Dec-2009 17:44
Send private message

re compensation:


we intend to compensate customers appropriately, (even though our Ts & Cs don't require us to), and we'll make that decision once everything is restored

 1 | 2
View this topic in a long page with up to 500 replies per page Create new topic



Twitter »

Follow us to receive Twitter updates when new discussions are posted in our forums:



Follow us to receive Twitter updates when news items and blogs are posted in our frontpage:



Follow us to receive Twitter updates when tech item prices are listed in our price comparison site:





News »

Intel unveils the 8th Gen Intel Core Processor family for desktop
Posted 25-Sep-2017 19:45


Chow brothers plan to invest NZ$100 million in technology
Posted 24-Sep-2017 16:24


Symantec protects data everywhere with Information Centric Security
Posted 21-Sep-2017 15:33


FUJIFILM introduces X-E3 mirrorless camera with wireless connectivity
Posted 18-Sep-2017 13:53


Vodafone announces new plans with bigger data bundles
Posted 15-Sep-2017 10:51


Skinny launches phone with support for te reo Maori
Posted 14-Sep-2017 08:39


If Vodafone dropping mail worries you, you’re doing online wrong
Posted 11-Sep-2017 13:54


Vodafone New Zealand deploy live 400 gigabit system
Posted 11-Sep-2017 11:07


OPPO camera phones now available at PB Tech
Posted 11-Sep-2017 09:56


Norton Wi-Fi Privacy — Easy, flawed VPN
Posted 11-Sep-2017 09:48


Lenovo reveals new ThinkPad A Series
Posted 8-Sep-2017 14:37


Huawei passes Apple for the first time to capture the second spot globally
Posted 8-Sep-2017 10:45


Vodafone initiative enhances te reo Maori pronunciation on Google Maps
Posted 8-Sep-2017 10:40


Voyager Internet expand local internet phone services company with Conversant acquisition
Posted 6-Sep-2017 18:27


NOW Expands in to Tauranga
Posted 5-Sep-2017 18:16



Geekzone Live »

Try automatic live updates from Geekzone directly in your browser, without refreshing the page, with Geekzone Live now.



Are you subscribed to our RSS feed? You can download the latest headlines and summaries from our stories directly to your computer or smartphone by using a feed reader.

Alternatively, you can receive a daily email with Geekzone updates.