Planned 2Degrees system outage

Forums › 2degrees (including Slingshot, Orcon, Flip, Stuff Fibre, MyRepublic, 2talk and Vocus) › Planned 2Degrees system outage

drajk

202 posts

Master Geek

Trusted

#208566 17-Feb-2017 01:07

I phoned 200 tonight and heard a message about a 2degrees system outage and thought I must have heard incorrectly when the message stated that hopefully systems would be functional by Wednesday (i.e. 6 days away).

Alas when I tried to login to the online website I learned that many systems were down and the link https://www.2degreesmobile.co.nz/help-and-support/mobile/service-status/#tab1

explained that many were going to be unavailable for approximately 1 week!

This amazes me in 2017. It suggests poor planning, cost minimisation with respect to upgrade resource, or at least a blatant disregard for customer service.

While there are some workarounds for some unavailable functions the idea that a supposedly serious mobile communications provider who has pitched itself as being a serious contender in the business marker can tell account customers that their balances and usage may not be accurate and may not update for a week is somewhat laughable.

At a minimum 2degrees should in my opinion be extending due dates by at least 10 days if they think a week of incorrect data is acceptable as customers should be given time to review their accounts once the system is back online and the data is updated.

Similarly for prepay customers - how are you supposed to find out your credit expiry date as neither the *100*1# nor BAL to 233 workarounds actually supply the credit expiry date. In a similar fashion 2degrees should really extend all prepay credit expiry dates which fall from 15.2.2017 to 25.2.2017 by ten days to allow prepay customers to determine when they need to topup. Why should people have to topup just because they suspect that their credit MAY be due to expire during the outage period.

I have previously developed, supported, and upgraded a significant healthcare IT system which had real-time patient information and was networked across multiple sites. Whenever I managed an upgrade I considered a 1hour outage as unacceptable - I could certainly have had more sleep if I had been able to take the system down for a week but this was impractical and to be honest would have been completely unnecessary as we always planned for minimum downtime. I have never seen any of the major Banks take their systems down for a week for planned upgrade and I really hope that 2degrees realises that such a lengthy upgrade is really unacceptable for customers and reviews its processes around such upgrades as the impression it gives is either that they are not a serious reliable service provider or that they don't consider customers when they 'plan' such lengthy outages.

1 | 2 | 3 | 4 | 5 | 6 | ... | 11

3918 posts

Uber Geek

Trusted

#1721639 17-Feb-2017 01:09

Woah yea that seems huge! It must be a massive change if they are going to have key systems offline for that long and I share your sentiment that it isn't really acceptable for a company in their position.

Speedtest 2019-10-14

drajk

202 posts

Master Geek

Trusted

#1721640 17-Feb-2017 01:12

For those that haven't seen 2degrees page re this -

copied from https://www.2degreesmobile.co.nz/help-and-support/mobile/service-status/#tab1

INTERNAL SYSTEMS UPGRADE: 2degrees is doing an internal systems upgrade from 15 to 22nd Feb.

This WON'T impact calling, text, data, fixed or broadband but unfortunately some of our mobile tools and options will be out of action.

Here's what you need to know:

From 15th to 22nd Feb

Online shop unavailable – for help with sign ups you can call our friendly Sales team on 0800 000 202 between 8am-8pm any day or pop into one of our physical stores https://www.2degreesmobile.co.nz/store-locator/.
You also won't be able to bring your mobile number to 2degrees over this period - you can still get a new 2degrees number and you’ll be able to bring your existing number after the 22nd.

From 16th to 22nd Feb:

2degrees online payment channels unavailable - you can still pay your bill via your Internet banking using your 2degrees account number as the reference.
2degrees mobile app / Your 2degrees unavailable – you can still top up and purchase value packs via text (prior to scheduled top-up outage below). To buy a Value Pack text ‘buy’ then the Value Pack name to 233, (i.e. ‘buy 19Carry’). To Top-Up text as per voucher instructions. Prepay customers can get their balance by dialling *100*1# or texting ‘BAL’ to 233.
Pay Monthly and Business balances may not be up to date over this period - but it will all be working in the back end and update after the outage.

From 21st (note this may change depending on schedule) to 22nd Feb:

Pre-pay top ups unavailable - so please top up in advance of the 21st and remember emergency calls can still be made.
Value pack purchase and changes unavailable.

We are aware this will cause customers some inconvenience and can only apologise. We’ve really done our best to minimise customer impacts but this is an important backend tech upgrade for 2degrees that will ultimately enable us to service you better.

Please bear with us – our teams will be working around the clock. Please also note that as planned and organised as we’ve tried to be, some of the above timings may be subject to change – but we will keep you in the loop on that.

The team at 2degrees

Riggleby

88 posts

Master Geek

#1721641 17-Feb-2017 03:14

It's really not that big of a deal. Seriously. Especially if your major gripe is balances not updating.

Aside from what you've mentioned, everything else is running as per normal (based on their list of what is down).

Similarly for prepay customers - how are you supposed to find out your credit expiry date as neither the *100*1# nor BAL to 233 workarounds actually supply the credit expiry date. In a similar fashion 2degrees should really extend all prepay credit expiry dates which fall from 15.2.2017 to 25.2.2017 by ten days to allow prepay customers to determine when they need to topup. Why should people have to topup just because they suspect that their credit MAY be due to expire during the outage period.

Um... What?

Prepay customers would not need to topup just because they assume they're low on credit - the system that tracks that stuff is still running (since it's not on their list), so they would still get the "You are low on credit/data/minutes" or "You need to topup to renew your pack" texts to indicate they need to do anything. There is no need to assume.

At a minimum 2degrees should in my opinion be extending due dates by at least 10 days if they think a week of incorrect data is acceptable as customers should be given time to review their accounts once the system is back online and the data is updated.

Again - um... what?

Given their billing is in arrears, any bills generated will be for the previous month that predates the maintenance period - therefore people will have been able to keep up-to-date with any additional charges in realtime. Even if they hadn't kept up-to-date themselves, the breakdown on the bills would provide enough information to explain where the extra charges came from.

I have previously developed, supported, and upgraded a significant healthcare IT system which had real-time patient information and was networked across multiple sites. Whenever I managed an upgrade I considered a 1hour outage as unacceptable - I could certainly have had more sleep if I had been able to take the system down for a week but this was impractical and to be honest would have been completely unnecessary as we always planned for minimum downtime. I have never seen any of the major Banks take their systems down for a week for planned upgrade and I really hope that 2degrees realises that such a lengthy upgrade is really unacceptable for customers and reviews its processes around such upgrades as the impression it gives is either that they are not a serious reliable service provider or that they don't consider customers when they 'plan' such lengthy outages.

Honestly, who cares?

They wouldn't have pulled all these things down for such a long period of time if it were avoidable. Whatever it is they're doing, they obviously weren't able to break it up into smaller stages and do it all during the dead of night when most wouldn't be affected.

drajk

202 posts

Master Geek

Trusted

#1721654 17-Feb-2017 07:35

Riggleby:

It's really not that big of a deal. Seriously. Especially if your major gripe is balances not updating.

Aside from what you've mentioned, everything else is running as per normal (based on their list of what is down).

Similarly for prepay customers - how are you supposed to find out your credit expiry date as neither the *100*1# nor BAL to 233 workarounds actually supply the credit expiry date. In a similar fashion 2degrees should really extend all prepay credit expiry dates which fall from 15.2.2017 to 25.2.2017 by ten days to allow prepay customers to determine when they need to topup. Why should people have to topup just because they suspect that their credit MAY be due to expire during the outage period.

Um... What?

Prepay customers would not need to topup just because they assume they're low on credit - the system that tracks that stuff is still running (since it's not on their list), so they would still get the "You are low on credit/data/minutes" or "You need to topup to renew your pack" texts to indicate they need to do anything. There is no need to assume.

At a minimum 2degrees should in my opinion be extending due dates by at least 10 days if they think a week of incorrect data is acceptable as customers should be given time to review their accounts once the system is back online and the data is updated.

Again - um... what?

Given their billing is in arrears, any bills generated will be for the previous month that predates the maintenance period - therefore people will have been able to keep up-to-date with any additional charges in realtime. Even if they hadn't kept up-to-date themselves, the breakdown on the bills would provide enough information to explain where the extra charges came from.

I have previously developed, supported, and upgraded a significant healthcare IT system which had real-time patient information and was networked across multiple sites. Whenever I managed an upgrade I considered a 1hour outage as unacceptable - I could certainly have had more sleep if I had been able to take the system down for a week but this was impractical and to be honest would have been completely unnecessary as we always planned for minimum downtime. I have never seen any of the major Banks take their systems down for a week for planned upgrade and I really hope that 2degrees realises that such a lengthy upgrade is really unacceptable for customers and reviews its processes around such upgrades as the impression it gives is either that they are not a serious reliable service provider or that they don't consider customers when they 'plan' such lengthy outages.

Honestly, who cares?

They wouldn't have pulled all these things down for such a long period of time if it were avoidable. Whatever it is they're doing, they obviously weren't able to break it up into smaller stages and do it all during the dead of night when most wouldn't be affected.

I would suggest that it is a good idea to read more carefully.

Addressing your reponses:

There is much that is not working other than just balances not updating - the '200' phone service, over the phone topups, the ability to top up other numbers, the online service and transaction lists ...

I did not say anything about running out of credit - my post referred to credit EXPIRY DATEs which cannot apparently be easily be confirmed by the workarounds left avaiable to customers - please reread if you don't understand.

A bill generated shortly after the outage may or may not be complete or accurate and with a limited period to pay there is limited time to review whether it is correct.

I care. I would be surprised if no one else does, especially those of us with multiple accounts. If all of the operational systems recording usage, maintaining balances to allow/disallow usage etc are still functioning correctly as is alleged then it would take a lot to convince me that the customer facing systems/reporting need to be down for a continuous one week period - to me the plan appears to be non customer-focused and suggestive that the upgrade is being done in a more cost effective way for the company rather than one which prioritises uptime and customer service.

Linux

11435 posts

Uber Geek

Trusted

Lifetime subscriber

#1721656 17-Feb-2017 07:57

Well the upgrade/s are happening and clearly it needs to be done

Linux

SaltyNZ

8241 posts

Uber Geek

Trusted

2degrees

Lifetime subscriber

#1721659 17-Feb-2017 08:08

drajk:

I have previously developed, supported, and upgraded a significant healthcare IT system which had real-time patient information and was networked across multiple sites. Whenever I managed an upgrade I considered a 1hour outage as unacceptable - I could certainly have had more sleep if I had been able to take the system down for a week but this was impractical and to be honest would have been completely unnecessary as we always planned for minimum downtime. I have never seen any of the major Banks take their systems down for a week for planned upgrade and I really hope that 2degrees realises that such a lengthy upgrade is really unacceptable for customers and reviews its processes around such upgrades as the impression it gives is either that they are not a serious reliable service provider or that they don't consider customers when they 'plan' such lengthy outages.

We also strive for minimum downtime. Unfortunately this process is not an upgrade, it is a wholesale migration which involves migrating the complete billing history of a million or so people as well as all the customer care channels and back-end provisioning from one system to a new and completely different one. I'm not familiar with healthcare IT systems, but telecommunications vendors don't typically build their systems to seamlessly work with two completely different business support systems at the same time, so some period of outage is unfortunately unavoidable.

drajk:

If all of the operational systems recording usage, maintaining balances to allow/disallow usage etc are still functioning correctly as is alleged then it would take a lot to convince me that the customer facing systems/reporting need to be down for a continuous one week period - to me the plan appears to be non customer-focused and suggestive that the upgrade is being done in a more cost effective way for the company rather than one which prioritises uptime and customer service.

At 2degrees, all of our customers are 100% real-time controlled. The system that measures and rates the usage and controls the network elements is not being replaced by this project. We will be upgrading some of its service logic during a very narrow window at the very end of the advertised period mainly so as to tweak the CDRs for the new billing system, but also to change the way notifications are handled, and to move from an old provisioning interface to a newer, more flexible one. As the developer of that service logic I have built a lot of safety valves into it to manage the transition so that there will be zero - not even an hour - planned interruption in its service availability to our customers.

However, billing and provisioning systems are a different kettle of fish. Having done basically this exact same project once before at a previous telco (not to mention that mobile is a very small industry and we all know the war stories about each other's upgrades, even the ones we weren't directly involved in) I assure you that there's no way around some outage. But also that this time round it's looking to be much quicker than the last one I worked on was.

We do apologise for the inconvenience.

iPad Pro 11" + iPhone 15 Pro Max + 2degrees 4tw!

These comments are my own and do not represent the opinions of 2degrees.

old3eyes

9120 posts

Uber Geek

Subscriber

#1721672 17-Feb-2017 08:56

Haven't seen anything in the media about this.. Oh wait. It's not Spark or Vodafone.

Regards,

Old3eyes

Sharesies

Trade NZ and US shares and funds with Sharesies (affiliate link).

drajk

202 posts

Master Geek

Trusted

#1721678 17-Feb-2017 09:10

SaltyNZ:

drajk:

I have previously developed, supported, and upgraded a significant healthcare IT system which had real-time patient information and was networked across multiple sites. Whenever I managed an upgrade I considered a 1hour outage as unacceptable - I could certainly have had more sleep if I had been able to take the system down for a week but this was impractical and to be honest would have been completely unnecessary as we always planned for minimum downtime. I have never seen any of the major Banks take their systems down for a week for planned upgrade and I really hope that 2degrees realises that such a lengthy upgrade is really unacceptable for customers and reviews its processes around such upgrades as the impression it gives is either that they are not a serious reliable service provider or that they don't consider customers when they 'plan' such lengthy outages.

We also strive for minimum downtime. Unfortunately this process is not an upgrade, it is a wholesale migration which involves migrating the complete billing history of a million or so people as well as all the customer care channels and back-end provisioning from one system to a new and completely different one. I'm not familiar with healthcare IT systems, but telecommunications vendors don't typically build their systems to seamlessly work with two completely different business support systems at the same time, so some period of outage is unfortunately unavoidable.

drajk:

If all of the operational systems recording usage, maintaining balances to allow/disallow usage etc are still functioning correctly as is alleged then it would take a lot to convince me that the customer facing systems/reporting need to be down for a continuous one week period - to me the plan appears to be non customer-focused and suggestive that the upgrade is being done in a more cost effective way for the company rather than one which prioritises uptime and customer service.

At 2degrees, all of our customers are 100% real-time controlled. The system that measures and rates the usage and controls the network elements is not being replaced by this project. We will be upgrading some of its service logic during a very narrow window at the very end of the advertised period mainly so as to tweak the CDRs for the new billing system, but also to change the way notifications are handled, and to move from an old provisioning interface to a newer, more flexible one. As the developer of that service logic I have built a lot of safety valves into it to manage the transition so that there will be zero - not even an hour - planned interruption in its service availability to our customers.

However, billing and provisioning systems are a different kettle of fish. Having done basically this exact same project once before at a previous telco (not to mention that mobile is a very small industry and we all know the war stories about each other's upgrades, even the ones we weren't directly involved in) I assure you that there's no way around some outage. But also that this time round it's looking to be much quicker than the last one I worked on was.

We do apologise for the inconvenience.

Thanks salty, I appreciate the magnitude of what you are doing. I also accept that "some period of outage is unavoidable" however I find it hard to believe that this should be 168+ hours. If the new platform has been thoroughly tested pre-migration and there are efficient methods of data transfer available between the two systems then that is a very long time even with a million customers and the complexity of different customer types/plans/packages. If the usage/control data is not affected why couldn't the 'old' online account viewing be updated at least daily during such a long outage with clear disclaimer that the data may be up to 24 hours old. I also think it is poor that the methods left available to prepay customers to check balances do not provide detail of credit expiry dates (happy to be proven wrong re this but I can't see how one obtains this information).

In any case everybody is entitled to their own opinion - I just find it surprising that such a long outage is considered acceptable.

MrCurly

10 posts

Wannabe Geek

#1722684 19-Feb-2017 14:58

I totally agree with D.

My background is also IT, including many years managing a well known AU Telcos Nonstop systems - this length of portal outage would not have been tolerated.

If 2degrees feels it is imperative to perform a data migration which has a known lengthy impact to customer services, then it would be common courtesy to send out a couple of SMS's during the preceding weeks, warning of the upcoming customer portal outage and advising to topup via the webpage if required.

Linux

11435 posts

Uber Geek

Trusted

Lifetime subscriber

#1722701 19-Feb-2017 16:04

Title of thread needs the word ' planned ' added

Linux

Stu

Hammered
8351 posts

Uber Geek

Moderator

ID Verified

Trusted

Lifetime subscriber

#1722709 19-Feb-2017 16:20

So will the automatic monthly pre-pay top-ups from a credit card happen? If not, will value packs roll over anyway to keep services working? What contingencies are in place to keep customers working if the upgrade falls over and top-ups don't go through? Will service be lost if you don't top-up with an alternative method?

Seems to me that with the systems that will be down, that 2Degrees really should have emailed and sent an SMS to every customer (or at least every customer likely to be affected).

People often mistake me for an adult because of my age.

Keep calm, and carry on posting.

Referral Links: Sharesies - Backblaze -

Are you happy with what you get from Geekzone? If so, please consider supporting us by subscribing.

No matter where you go, there you are.

Stu

Hammered
8351 posts

Uber Geek

Moderator

ID Verified

Trusted

Lifetime subscriber

#1722712 19-Feb-2017 16:25

Linux: Title of thread needs the word ' planned ' added

Linux

Done, although the fact it was planned should call for customers to be proactively advised in advance, and not just an entry on the Service status page!

People often mistake me for an adult because of my age.

Keep calm, and carry on posting.

Referral Links: Sharesies - Backblaze -

Are you happy with what you get from Geekzone? If so, please consider supporting us by subscribing.

No matter where you go, there you are.

toejam316

1466 posts

Uber Geek

Trusted

Lifetime subscriber

#1722770 19-Feb-2017 19:06

Agreed with above - I understand Telco systems are nasty, multi-headed beasts, but best practice for a known national outage would be some sort of notifications prior to the service outage period, not while it is in progress. It's not like 2degrees doesn't know how to get in touch with their customers.

Anything I say is the ramblings of an ill informed, opinionated so-and-so, and not representative of any of my past, present or future employers, and is also probably best disregarded.

DjShadow

4093 posts

Uber Geek

ID Verified

Trusted

#1722780 19-Feb-2017 19:44

From memory didn't Spark have an outage for just about as long a while back?

If so then this is far from a unique situation

catfood

4 posts

Wannabe Geek

#1724353 22-Feb-2017 11:59

I haven't been able to top up my phone balance for five days now! Why wasn't any warning given?

If 2DegreesMobile were actually sincere about customer inconvenience then they would have provided free or debt/IOU calling to prepaid customers whose balances expired and couldn't be topped up during the planned upgrade period, especially when they didn't provide any forewarning at all, just a technical web page notification which nobody would normally see.

What an incredibly arrogant lapse in customer service to not provide any forewarning via sms and media outlets when such a long period of disrupted service was planned. Sure, this may well have been a huge and necessary upgrade but its roll out has been a joke and the only conclusion that can be drawn from this experience this that 2Degreemobile clearly don't feel any sense of responsibility or goodwill towards their prepaid customers.

1 | 2 | 3 | 4 | 5 | 6 | ... | 11