My phone just registered again...

The lack of information is really annoying. My parents ADSL has been down pretty-much all day.

From Here:

An update for Vodafone (previously WorldxChange Communication) customers:
(4:45pm)

Earlier today, an electricity outage at one of the Auckland operations centres resulted in a loss of broadband and/or landline services for a group of consumer and business customers. Our team has been working hard to restore services. Throughout the afternoon and early evening, customers will be progressively brought back on line. We expect all customers to be returned to normal service this evening. We sincerely apologise for any inconvenience caused.

For further information, customers can contact us by any of the following channels:
Customer Care - 777 or 0800 800 021
Customer Care (Enterprise and Business Customers) - 888 or 0800 400 888
Vodafone Community - http://community.vodafone.co.nz
Facebook - facebook.com/vodafoneNZ/
Twitter -twitter.com/vodafoneNZ
Many Thanks,
Vodafone Operations

Now, why the hell did they have a power outage? In almost every DC there are generators and backups for almost everything and monthly (or even weekly) tests are done to ensure those backups are still running. This does really sound catastrophic to cause such a widespread outage.

stevenb: Vodafones Status update page

404

on 16m ago

A further update for Vodafone (previously WorldxChange Communication) customers:

(6:00pm)

Earlier today, an electricity outage at one of the Auckland operations centres resulted in a loss of broadband and/or landline services for a group of consumer and business customers.

We can now advise operations have been restored and within the next hour customers should be back to normal service for both broadband and landlines.

Some residential customers may need to reboot their modem to restore the broadband connection.

Thank you for your patience and apologies once again for any inconvenience,

Vodafone Operations

Azzaw1980: Anyone having authentication problems when attempting to download emails to.a xnet address? Our phone service is all back to normal but the email won't authenticate. This is all after the outage. Cheers

I'd wait a little longer - looks like VFX may be up but xDSL is not authenticating for everyone yet. No doubt systems are being bought back up one at a time.

DNS has come back but their website is still offline, so yeah I'd say they still booting systems.

I'm sure many systems may have not been rebooted for years so may be going through FSCK (file system checks).

I see their Auckland router only came back online 50 minutes ago when we re-established BGP

xDSL up now?

icepicknz:

 

DNS has come back but their website is still offline, so yeah I'd say they still booting systems.

 

I'm sure many systems may have not been rebooted for years so may be going through FSCK (file system checks).

 

I see their Auckland router only came back online 50 minutes ago when we re-established BGP

 

even if they had long fsck times you'd think by now that their web site would be back up.

maybe they're restoring from tape backup due to raid failures?

What a comedy to read all this lot.  Your complaints and frustration make very amusing reading.  Obviously very few of the posters here have ever worked in a team responsible for some form of public service.  If a major fuse blows you don't go looking for generators - that could run the risk of further damage.  You have to establish why the fuse blew and remove the fault if possible.  You have to ensure that nobody was injured by the exploding fuse - and render first aid, call ambulance etc even before starting on restoring service. Then you have to restore the service which may mean reloading systems, going through required warm up times etc.  Checking all the way that the systems themselves weren't damaged by the incident that blew the fuse.  You don't man/people your phones because these days they're probably run off the same power source that just got zapped.  If you have additional spare staff not actively engaged in restoring the service, then maybe, just maybe, you get them involved in trying to explain it all to several thousand users - provided that you have some way of communicating with them other than by waving a long banner out of the window.  I don't know anything about WXC installations, I'm just guessing. And probably weekend means reduced staff levels because the rate you pay for service doesn't cover 24x7 full staffing.

Fellow Geeks -if your activities require absolutely 100% failsafe phone and internet services then you're going to have to face up to some hefty bills for that level of service, if you can find a provider willing to take it on.

crichton:

 

What a comedy to read all this lot.  Your complaints and frustration make very amusing reading.  Obviously very few of the posters here have ever worked in a team responsible for some form of public service.  If a major fuse blows you don't go looking for generators - that could run the risk of further damage.  You have to establish why the fuse blew and remove the fault if possible.  You have to ensure that nobody was injured by the exploding fuse - and render first aid, call ambulance etc even before starting on restoring service. Then you have to restore the service which may mean reloading systems, going through required warm up times etc.  Checking all the way that the systems themselves weren't damaged by the incident that blew the fuse.  You don't man/people your phones because these days they're probably run off the same power source that just got zapped.  If you have additional spare staff not actively engaged in restoring the service, then maybe, just maybe, you get them involved in trying to explain it all to several thousand users - provided that you have some way of communicating with them other than by waving a long banner out of the window.  I don't know anything about WXC installations, I'm just guessing. And probably weekend means reduced staff levels because the rate you pay for service doesn't cover 24x7 full staffing.

 

Fellow Geeks -if your activities require absolutely 100% failsafe phone and internet services then you're going to have to face up to some hefty bills for that level of service, if you can find a provider willing to take it on.

 

it does surprise me when "reasonable" sized companies don't even have redundant DNS infrastructure across multiple locations.

i seem to recall callplus having a wide-affecting outage but for less duration a while back. (but before the orcon/m2 merger)

if they at least had some redundant DNS they could host a web page somewhere giving network status at the least.

and it's not like the support staff are necessarily going to be able to do a lot for getting things back up and going, so they may be able to at least update network status or give an estimated time for restoration of services.  especially when it's something as far reaching as phone systems.

it's easy to have two internet connections.  it's not easy to have two phone numbers.

crichton:  If you have additional spare staff not actively engaged in restoring the service, then maybe, just maybe, you get them involved in trying to explain it all to several thousand users - provided that you have some way of communicating with them other than by waving a long banner out of the window.  I don't know anything about WXC installations, I'm just guessing. And probably weekend means reduced staff levels because the rate you pay for service doesn't cover 24x7 full staffing.

Ahhh, but if it takes 1 staff member, 15 minutes out of queue/off task to bring attention to people that can (or in a small organization, do it themselves) post about the issue properly on Social Media/backup sites, then you can save hours of hell later on (customers threatening to walk, wanting even more details, etc etc), and save company-face.  15 minutes to communicate with your smart/switched-on customers can be worth a lot of face in this world where people can vote with their feet at the drop of a hat.

My 2c.