Geekzone: technology news, blogs, forums
Guest
Welcome Guest.
You haven't logged in yet. If you don't have an account you can register now.




253 posts

Ultimate Geek

Geekstore

Topic # 54440 17-Dec-2009 20:27
Send private message

Could just be me, but my cloud server is down, and so is the control panel at https://console.iservices.net.nz/

Also found out that iServe have no after-hours support for incidents like this.. Wonderful.

So, just me, or is it everyone?




Shaun Fisher - www.geekstore.co.nz

E: shaun[at]geekstore[dot]co[dot]nz
P: 0800894508  F: 0800897451

View this topic in a long page with up to 500 replies per page Create new topic
 1 | 2
1200 posts

Uber Geek
+1 received by user: 3

Trusted

  Reply # 283530 17-Dec-2009 20:42
Send private message


Down for me as well. We were only trialing their service but have my doubts now, I can understand a bit of downtime but no after hours support? Do they wait until 8AM to resolve? Hmmmmmmmmmm.

I wonder how long they will be offline for.




Tyler - Parnell Geek - iPhone 3G - Lenovo X301 - Kaseya - Great Western Steak House, these are some of my favourite things.



253 posts

Ultimate Geek

Geekstore

  Reply # 283534 17-Dec-2009 20:51
Send private message

I find the fact that there's no after-hours support for not only cloud computing, but anything within their network to be quite a let down to be honest.




Shaun Fisher - www.geekstore.co.nz

E: shaun[at]geekstore[dot]co[dot]nz
P: 0800894508  F: 0800897451

 
 
 
 


566 posts

Ultimate Geek
+1 received by user: 73


  Reply # 283543 17-Dec-2009 21:35
Send private message

From http://www.iserve.co.nz/announcements.php

Thursday 17 December 20:18
Due to current issues with our cloud and virtual server environment we have disabled administrative access to all cloud and virtual machines.

Technicians are currently investigating and hope to have a resolution within the next 24 hours.

6294 posts

Uber Geek
+1 received by user: 377

Moderator
Trusted
Subscriber

  Reply # 283545 17-Dec-2009 21:47
Send private message

Yeah I can't access the control panel either, good thing is our servers are still up.

1200 posts

Uber Geek
+1 received by user: 3

Trusted

  Reply # 283548 17-Dec-2009 21:51
Send private message

Kraven: From http://www.iserve.co.nz/announcements.php

Thursday 17 December 20:18
Due to current issues with our cloud and virtual server environment we have disabled administrative access to all cloud and virtual machines.

Technicians are currently investigating and hope to have a resolution within the next 24 hours.


So it looks like they DO have 24 hour monitoring, but 24 hours! Lucky the cloud is still up.

New Zealand's first shaky steps into the cloud.




Tyler - Parnell Geek - iPhone 3G - Lenovo X301 - Kaseya - Great Western Steak House, these are some of my favourite things.



253 posts

Ultimate Geek

Geekstore

  Reply # 283553 17-Dec-2009 22:07
Send private message

Kraven: From http://www.iserve.co.nz/announcements.php

Thursday 17 December 20:18
Due to current issues with our cloud and virtual server environment we have disabled administrative access to all cloud and virtual machines.

Technicians are currently investigating and hope to have a resolution within the next 24 hours.


Seems that they posted that at the same time I phoned up Orcon.. Hrm.... Undecided




Shaun Fisher - www.geekstore.co.nz

E: shaun[at]geekstore[dot]co[dot]nz
P: 0800894508  F: 0800897451

1 post

Wannabe Geek


  Reply # 283856 18-Dec-2009 18:20
Send private message

Well its now been 24 hours ...My clould server is still floating down river an Iserve have yet to explain why other than technical issues and some bizzare reference to the fact that the admin interface has been disabled and the fact they have had to deploy new hardware

"Our technicians continue to work on a resolution to the issues we have with access to our cloud and VM platforms and are now deploying new hardware to assist in a resolution. We regret the length of time that access to these servers is taking, and deeply apologise for the inconvenience that this is causing our customers"



Phone calls to the helpdesk go along the lines of :

What is wrong ?- We dont know we are not technical
When will it be available - We dont know
Can I speak with someone about it - There is no one available
Can I speak to a manager about it - There are none available they will call you back (they dont)

I have also had -

"All the technical team have left you have called after hours" at 10 mins to closing

"We tried to call you but your number does not work" - All my customers complaining dont seem to have any issue calling me on it 

"If you emailed us we would have replied" ...I did twice and they didnt

Your email took 15 mins to get to us - Perhaps because I used your servers  



Orcon if your listening - What a lemon you have brought !!

6294 posts

Uber Geek
+1 received by user: 377

Moderator
Trusted
Subscriber

  Reply # 283870 18-Dec-2009 19:09
Send private message

platinumit: Well its now been 24 hours ...My clould server is still floating down river an Iserve have yet to explain why other than technical issues and some bizzare reference to the fact that the admin interface has been disabled and the fact they have had to deploy new hardware


I saw your comments that came through on my blog post, thanks for that.

Our servers are still up, console is still down, very concerning considering this is coming up 48 hours.

What's going on iServe?

8 posts

Wannabe Geek

Trusted
iSERVE Ltd

  Reply # 283981 19-Dec-2009 11:31
Send private message

Update on iSERVE Cloud and VM platform access.

On Thursday 17 December, while undertaking routine administration, the storage hosting the cloud virtual machines was found to have a corrupt partition table.

Steps were undertaken after to remedy the problem. At this time access to administer VM and Cloud servers was prohibited to ensure that no service was restarted or shut down, which may have resulted in unrecoverable data loss. Servers remained up and stable during this time.

On the 18th of December, we commenced deploying our secondary SAN as the primary SAN for this platform, and performed a backup of the physical drives from the original san to ensure we could restore files in a worst case scenario . This process did not complete until 3.30am.

At 4am 19th December, We powered off all machines in order to enable us to commence a disk to device backup to the new SAN. It was estimated that the total down time would be around 4.5 hours, however this process is taking longer than expected. We now expect this process to be finished at approximately 4pm this afternoon, at which time we will restore the partition table and metadata backups to restore the partitions and logical volumes. We will advise further once the data has been restored to the new storage device.

1200 posts

Uber Geek
+1 received by user: 3

Trusted

  Reply # 283986 19-Dec-2009 11:51
Send private message

iserve: Update on iSERVE Cloud and VM platform access.

On Thursday 17 December, while undertaking routine administration, the storage hosting the cloud virtual machines was found to have a corrupt partition table.

Steps were undertaken after to remedy the problem. At this time access to administer VM and Cloud servers was prohibited to ensure that no service was restarted or shut down, which may have resulted in unrecoverable data loss. Servers remained up and stable during this time....


First off thankyou for the update and to the geeks working hard on this. It seems to be you have it in hand and know what you are doing, the corruption happening in the first place on the SAN is a worry but these things happen I suppose.

The real worry and the thing which is majorly freaking people out is your esculation paths for the cloud seem be shuffling through a normal ISP helpdesk which no one has a clue, so even if you have all this going on everyone thinks no one is working on it and have all gone home. Perhaps time to get a dedicated helpdesk, or keep the current one up with the play.

Thanks again for posting on geekzone, and good luck, your service looks good and I have been trialing it. The SAN corruption does worry us through.




Tyler - Parnell Geek - iPhone 3G - Lenovo X301 - Kaseya - Great Western Steak House, these are some of my favourite things.

8 posts

Wannabe Geek

Trusted
iSERVE Ltd

  Reply # 283993 19-Dec-2009 12:18
Send private message

exportgoldman:
The real worry and the thing which is majorly freaking people out is your esculation paths for the cloud seem be shuffling through a normal ISP helpdesk which no one has a clue, so even if you have all this going on everyone thinks no one is working on it and have all gone home. Perhaps time to get a dedicated helpdesk, or keep the current one up with the play.

Thanks again for posting on geekzone, and good luck, your service looks good and I have been trialing it. The SAN corruption does worry us through.



Support for the prouduct is via our normal support channels published here:http://www.iserve.co.nz/contactus.php


We do have a dedicated helpdesk team available via the iserve contact details and I am aware that platinum have experienced an issue due to high call volumes.  When the call volume is high, we overflow our inbound calls to an external call centre.  The operator that Platinum spoke to advised him incorrectly of our support levels, staff are on site untill 7pm, and we monitor the services 24/7 - yes techs were here and working on it, unfortunately ( lets call it murphys law) Help Desk numbers were down due to illness - and we are loathed to put senior technicians on the phones to answer customers questions, when their skills are required to resolve the issue.  Obviously the time spent on the phone, is time not spent on resolving the issue.


On a general note, the iSERVE hosting support team has been light on resource for quite some time, however that has been addressed, and an improvement will be noticeable shortly.


iSERVE's support team were up to date with the progress, and Orcons support team were informed of the issues, however they would not have been able to offer any technical update or assistance with the problem - we didn't anticipate customers to be directing their inquiries to Orcon, and is something we will put consideration too for the future.


With regards to the data corruption, we believe that it is a freak incident, and we are all very lucky that the systems administrators we have were diligent enough to discover it before the issue became more serious. We are of course in communication with Citrix, and will continue to discuss it with them.


I hope the above is legible - like all our techs, there has been very little sleep had over the past 2 nights

6294 posts

Uber Geek
+1 received by user: 377

Moderator
Trusted
Subscriber

  Reply # 283999 19-Dec-2009 13:23
Send private message

Just spoke to the iServe ops manager, hopefully servers will be back up at 4pm.

/fingers crossed.



253 posts

Ultimate Geek

Geekstore

  Reply # 284051 19-Dec-2009 17:27
Send private message

---update--- 4.59 19 December
The backup process continues, with close to 90% of the process complete, we now estimate it will finish at approx 6.45pm

Source: http://www.iserve.co.nz/announcements.php




Shaun Fisher - www.geekstore.co.nz

E: shaun[at]geekstore[dot]co[dot]nz
P: 0800894508  F: 0800897451



253 posts

Ultimate Geek

Geekstore

  Reply # 284054 19-Dec-2009 17:39
Send private message

@iserve

Will we be getting any sort of service credit for this trouble?




Shaun Fisher - www.geekstore.co.nz

E: shaun[at]geekstore[dot]co[dot]nz
P: 0800894508  F: 0800897451

8 posts

Wannabe Geek

Trusted
iSERVE Ltd

  Reply # 284055 19-Dec-2009 17:44
Send private message

re compensation:


we intend to compensate customers appropriately, (even though our Ts & Cs don't require us to), and we'll make that decision once everything is restored

 1 | 2
View this topic in a long page with up to 500 replies per page Create new topic



Twitter »

Follow us to receive Twitter updates when new discussions are posted in our forums:



Follow us to receive Twitter updates when news items and blogs are posted in our frontpage:



Follow us to receive Twitter updates when tech item prices are listed in our price comparison site:





News »

Phone prices rising as users move upmarket
Posted 24-Nov-2017 17:16


Talking net neutrality on RNZ Nine-to-Noon
Posted 24-Nov-2017 12:11


Air New Zealand experiments with blockchain technology
Posted 23-Nov-2017 15:39


Symantec selects Amazon Web Services to deliver cloud security
Posted 23-Nov-2017 10:40


New Zealand Ministry of Education chooses Unisys for cloud-based education resourcing management system
Posted 22-Nov-2017 22:00


Business analytics software powers profits for NZ wine producers
Posted 22-Nov-2017 21:52


Pyrios strikes up alliance with Microsoft integrator UC Logiq
Posted 22-Nov-2017 21:51


The New Zealand IT services ecosystem - it's all digital down here
Posted 22-Nov-2017 21:49


Volvo to supply tens of thousands of autonomous drive compatible cars to Uber
Posted 22-Nov-2017 21:46


From small to medium and beyond: Navigating the ERP battlefield
Posted 21-Nov-2017 21:12


Business owners: ERP software selection starts (and finishes) with you
Posted 21-Nov-2017 21:11


Why I'm not an early adopter
Posted 21-Nov-2017 10:39


Netatmo launches smart home products in New Zealand
Posted 20-Nov-2017 20:06


Huawei Mate 10: Punchy, long battery life, artificial intelligence
Posted 20-Nov-2017 16:30


Propel launch Disney Star Wars Laser Battle Drones
Posted 19-Nov-2017 21:26



Geekzone Live »

Try automatic live updates from Geekzone directly in your browser, without refreshing the page, with Geekzone Live now.



Are you subscribed to our RSS feed? You can download the latest headlines and summaries from our stories directly to your computer or smartphone by using a feed reader.

Alternatively, you can receive a daily email with Geekzone updates.