Geekzone: technology news, blogs, forums
Guest
Welcome Guest.
You haven't logged in yet. If you don't have an account you can register now.




188 posts

Master Geek
+1 received by user: 15

Trusted

# 208566 17-Feb-2017 01:07
One person supports this post
Send private message

I phoned 200 tonight and heard a message about a 2degrees system outage and thought I must have heard incorrectly when the message stated that hopefully systems would be functional by Wednesday (i.e. 6 days away).

 

Alas when I tried to login to the online website I learned that many systems were down and the link https://www.2degreesmobile.co.nz/help-and-support/mobile/service-status/#tab1

 

explained that many were going to be unavailable for approximately 1 week!

 

This amazes me in 2017. It suggests poor planning, cost minimisation with respect to upgrade resource, or at least a blatant disregard for customer service.

 

While there are some workarounds for some unavailable functions the idea that a supposedly serious mobile communications provider who has pitched itself as being a serious contender in the business marker can tell account customers that their balances and usage may not be accurate and may not update for a week is somewhat laughable.

 

At a minimum 2degrees should in my opinion be extending due dates by at least 10 days if they think a week of incorrect data is acceptable as customers should be given time to review their accounts once the system is back online and the data is updated.

 

Similarly for prepay customers - how are you supposed to find out your credit expiry date as neither the *100*1# nor BAL to 233 workarounds actually supply the credit expiry date. In a similar fashion 2degrees should really extend all prepay credit expiry dates which fall from 15.2.2017 to 25.2.2017 by ten days to allow prepay customers to determine when they need to topup. Why should people have to topup just because they suspect that their credit MAY be due to expire during the outage period.

 

I have previously developed, supported, and upgraded a significant healthcare IT system which had real-time patient information and was networked across multiple sites. Whenever I managed an upgrade I considered a 1hour outage as unacceptable - I could certainly have had more sleep if I had been able to take the system down for a week but this was impractical and to be honest would have been completely unnecessary as we always planned for minimum downtime. I have never seen any of the major Banks take their systems down for a week for planned upgrade and I really hope that 2degrees realises that such a lengthy upgrade is really unacceptable for customers and reviews its processes around such upgrades as the impression it gives is either that they are not a serious reliable service provider or that they don't consider customers when they 'plan' such lengthy outages.

 

 


View this topic in a long page with up to 500 replies per page Create new topic
 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11
3449 posts

Uber Geek
+1 received by user: 451

Trusted

  # 1721639 17-Feb-2017 01:09
Send private message

Woah yea that seems huge! It must be a massive change if they are going to have key systems offline for that long and I share your sentiment that it isn't really acceptable for a company in their position.








188 posts

Master Geek
+1 received by user: 15

Trusted

  # 1721640 17-Feb-2017 01:12
Send private message

For those that haven't seen 2degrees page re this - 

 

copied from https://www.2degreesmobile.co.nz/help-and-support/mobile/service-status/#tab1

 

     

  • MOBILE NETWORK
  • BROADBAND NETWORK

 

 

 

INTERNAL SYSTEMS UPGRADE:  2degrees is doing an internal systems upgrade from 15 to 22nd Feb.

 

This WON'T impact calling, text, data, fixed or broadband but unfortunately some of our mobile tools and options will be out of action. 

 

 

 

Here's what you need to know: 

 

From 15th to 22nd Feb

 

  • Online shop unavailable – for help with sign ups you can call our friendly Sales team on 0800 000 202 between 8am-8pm any day or pop into one of our physical stores https://www.2degreesmobile.co.nz/store-locator/.
  • You also won't be able to bring your mobile number to 2degrees over this period - you can still get a new 2degrees number and you’ll be able to bring your existing number after the 22nd.

From 16th to 22nd Feb:

 

  • 2degrees online payment channels unavailable - you can still pay your bill via your Internet banking using your 2degrees account number as the reference.
  • 2degrees mobile app / Your 2degrees unavailable – you can still top up and purchase value packs via text (prior to scheduled top-up outage below).  To buy a Value Pack text ‘buy’ then the Value Pack name to 233, (i.e. ‘buy 19Carry’). To Top-Up text as per voucher instructions. Prepay customers can get their balance by dialling *100*1# or texting ‘BAL’ to 233.
  • Pay Monthly and Business balances may not be up to date over this period - but it will all be working in the back end and update after the outage.

 

 

From 21st (note this may change depending on schedule) to 22nd Feb:

 

  • Pre-pay top ups unavailable - so please top up in advance of the 21st and remember emergency calls can still be made.
  • Value pack purchase and changes unavailable.

 

 

We are aware this will cause customers some inconvenience and can only apologise. We’ve really done our best to minimise customer impacts but this is an important backend tech upgrade for 2degrees that will ultimately enable us to service you better. 

 

 

 

Please bear with us – our teams will be working around the clock.  Please also note that as planned and organised as we’ve tried to be, some of the above timings may be subject to change – but we will keep you in the loop on that.

 

 

 

The team at 2degrees

 

 

 


 
 
 
 


76 posts

Master Geek
+1 received by user: 16


  # 1721641 17-Feb-2017 03:14
One person supports this post
Send private message

It's really not that big of a deal. Seriously. Especially if your major gripe is balances not updating.

 

Aside from what you've mentioned, everything else is running as per normal (based on their list of what is down).

 

Similarly for prepay customers - how are you supposed to find out your credit expiry date as neither the *100*1# nor BAL to 233 workarounds actually supply the credit expiry date. In a similar fashion 2degrees should really extend all prepay credit expiry dates which fall from 15.2.2017 to 25.2.2017 by ten days to allow prepay customers to determine when they need to topup. Why should people have to topup just because they suspect that their credit MAY be due to expire during the outage period.

 

Um... What?

 

Prepay customers would not need to topup just because they assume they're low on credit - the system that tracks that stuff is still running (since it's not on their list), so they would still get the "You are low on credit/data/minutes" or "You need to topup to renew your pack" texts to indicate they need to do anything. There is no need to assume.

 

At a minimum 2degrees should in my opinion be extending due dates by at least 10 days if they think a week of incorrect data is acceptable as customers should be given time to review their accounts once the system is back online and the data is updated.

 

Again - um... what?

 

Given their billing is in arrears, any bills generated will be for the previous month that predates the maintenance period - therefore people will have been able to keep up-to-date with any additional charges in realtime. Even if they hadn't kept up-to-date themselves, the breakdown on the bills would provide enough information to explain where the extra charges came from.

 

I have previously developed, supported, and upgraded a significant healthcare IT system which had real-time patient information and was networked across multiple sites. Whenever I managed an upgrade I considered a 1hour outage as unacceptable - I could certainly have had more sleep if I had been able to take the system down for a week but this was impractical and to be honest would have been completely unnecessary as we always planned for minimum downtime. I have never seen any of the major Banks take their systems down for a week for planned upgrade and I really hope that 2degrees realises that such a lengthy upgrade is really unacceptable for customers and reviews its processes around such upgrades as the impression it gives is either that they are not a serious reliable service provider or that they don't consider customers when they 'plan' such lengthy outages.

 

Honestly, who cares?

 

They wouldn't have pulled all these things down for such a long period of time if it were avoidable. Whatever it is they're doing, they obviously weren't able to break it up into smaller stages and do it all during the dead of night when most wouldn't be affected.




188 posts

Master Geek
+1 received by user: 15

Trusted

  # 1721654 17-Feb-2017 07:35
Send private message

Riggleby:

 

It's really not that big of a deal. Seriously. Especially if your major gripe is balances not updating.

 

Aside from what you've mentioned, everything else is running as per normal (based on their list of what is down).

 

Similarly for prepay customers - how are you supposed to find out your credit expiry date as neither the *100*1# nor BAL to 233 workarounds actually supply the credit expiry date. In a similar fashion 2degrees should really extend all prepay credit expiry dates which fall from 15.2.2017 to 25.2.2017 by ten days to allow prepay customers to determine when they need to topup. Why should people have to topup just because they suspect that their credit MAY be due to expire during the outage period.

 

Um... What?

 

Prepay customers would not need to topup just because they assume they're low on credit - the system that tracks that stuff is still running (since it's not on their list), so they would still get the "You are low on credit/data/minutes" or "You need to topup to renew your pack" texts to indicate they need to do anything. There is no need to assume.

 

At a minimum 2degrees should in my opinion be extending due dates by at least 10 days if they think a week of incorrect data is acceptable as customers should be given time to review their accounts once the system is back online and the data is updated.

 

Again - um... what?

 

Given their billing is in arrears, any bills generated will be for the previous month that predates the maintenance period - therefore people will have been able to keep up-to-date with any additional charges in realtime. Even if they hadn't kept up-to-date themselves, the breakdown on the bills would provide enough information to explain where the extra charges came from.

 

I have previously developed, supported, and upgraded a significant healthcare IT system which had real-time patient information and was networked across multiple sites. Whenever I managed an upgrade I considered a 1hour outage as unacceptable - I could certainly have had more sleep if I had been able to take the system down for a week but this was impractical and to be honest would have been completely unnecessary as we always planned for minimum downtime. I have never seen any of the major Banks take their systems down for a week for planned upgrade and I really hope that 2degrees realises that such a lengthy upgrade is really unacceptable for customers and reviews its processes around such upgrades as the impression it gives is either that they are not a serious reliable service provider or that they don't consider customers when they 'plan' such lengthy outages.

 

Honestly, who cares?

 

They wouldn't have pulled all these things down for such a long period of time if it were avoidable. Whatever it is they're doing, they obviously weren't able to break it up into smaller stages and do it all during the dead of night when most wouldn't be affected.

 

 

I would suggest that it is a good idea to read more carefully.

 

Addressing your reponses:

 

There is much that is not working other than just balances not updating - the '200' phone service, over the phone topups, the ability to top up other numbers, the online service and transaction lists ...

 

I did not say anything about running out of credit - my post referred to credit EXPIRY DATEs which cannot apparently be easily be confirmed by the workarounds left avaiable to customers - please reread if you don't understand.

 

A bill generated shortly after the outage may or may not be complete or accurate and with a limited period to pay there is limited time to review whether it is correct.

 

I care. I would be surprised if no one else does, especially those of us with multiple accounts. If all of the operational systems recording usage, maintaining balances to allow/disallow usage etc are still functioning correctly as is alleged then it would take a lot to convince me that the customer facing systems/reporting need to be down for a continuous one week period - to me the plan appears to be non customer-focused and suggestive that the upgrade is being done in a more cost effective way for the company rather than one which prioritises uptime and customer service.


4517 posts

Uber Geek
+1 received by user: 2623

Trusted
Lifetime subscriber

  # 1721656 17-Feb-2017 07:57
Send private message

Well the upgrade/s are happening and clearly it needs to be done

Linux



4889 posts

Uber Geek
+1 received by user: 2390

Trusted
Subscriber

  # 1721659 17-Feb-2017 08:08
3 people support this post
Send private message

drajk:

 

I have previously developed, supported, and upgraded a significant healthcare IT system which had real-time patient information and was networked across multiple sites. Whenever I managed an upgrade I considered a 1hour outage as unacceptable - I could certainly have had more sleep if I had been able to take the system down for a week but this was impractical and to be honest would have been completely unnecessary as we always planned for minimum downtime. I have never seen any of the major Banks take their systems down for a week for planned upgrade and I really hope that 2degrees realises that such a lengthy upgrade is really unacceptable for customers and reviews its processes around such upgrades as the impression it gives is either that they are not a serious reliable service provider or that they don't consider customers when they 'plan' such lengthy outages.

 

 

 

 

We also strive for minimum downtime. Unfortunately this process is not an upgrade, it is a wholesale migration which involves migrating the complete billing history of a million or so people as well as all the customer care channels and back-end provisioning from one system to a new and completely different one. I'm not familiar with healthcare IT systems, but telecommunications vendors don't typically build their systems to seamlessly work with two completely different business support systems at the same time, so some period of outage is unfortunately unavoidable.

 

 

 

drajk:

 

If all of the operational systems recording usage, maintaining balances to allow/disallow usage etc are still functioning correctly as is alleged then it would take a lot to convince me that the customer facing systems/reporting need to be down for a continuous one week period - to me the plan appears to be non customer-focused and suggestive that the upgrade is being done in a more cost effective way for the company rather than one which prioritises uptime and customer service.

 

 

 

 

At 2degrees, all of our customers are 100% real-time controlled. The system that measures and rates the usage and controls the network elements is not being replaced by this project. We will be upgrading some of its service logic during a very narrow window at the very end of the advertised period mainly so as to tweak the CDRs for the new billing system, but also to change the way notifications are handled, and to move from an old provisioning interface to a newer, more flexible one. As the developer of that service logic I have built a lot of safety valves into it to manage the transition so that there will be zero - not even an hour - planned interruption in its service availability to our customers.

 

However, billing and provisioning systems are a different kettle of fish. Having done basically this exact same project once before at a previous telco (not to mention that mobile is a very small industry and we all know the war stories about each other's upgrades, even the ones we weren't directly involved in) I assure you that there's no way around some outage. But also that this time round it's looking to be much quicker than the last one I worked on was.

 

We do apologise for the inconvenience.





iPad Pro 11" + iPhone XS + 2degrees 4tw!

 

These comments are my own and do not represent the opinions of 2degrees.


8012 posts

Uber Geek
+1 received by user: 835

Subscriber

  # 1721672 17-Feb-2017 08:56
Send private message

Haven't seen anything in the media about this..  Oh wait.  It's not Spark or Vodafone.





Regards,

Old3eyes




188 posts

Master Geek
+1 received by user: 15

Trusted

  # 1721678 17-Feb-2017 09:10
One person supports this post
Send private message

SaltyNZ:

 

drajk:

 

I have previously developed, supported, and upgraded a significant healthcare IT system which had real-time patient information and was networked across multiple sites. Whenever I managed an upgrade I considered a 1hour outage as unacceptable - I could certainly have had more sleep if I had been able to take the system down for a week but this was impractical and to be honest would have been completely unnecessary as we always planned for minimum downtime. I have never seen any of the major Banks take their systems down for a week for planned upgrade and I really hope that 2degrees realises that such a lengthy upgrade is really unacceptable for customers and reviews its processes around such upgrades as the impression it gives is either that they are not a serious reliable service provider or that they don't consider customers when they 'plan' such lengthy outages.

 

 

 

 

We also strive for minimum downtime. Unfortunately this process is not an upgrade, it is a wholesale migration which involves migrating the complete billing history of a million or so people as well as all the customer care channels and back-end provisioning from one system to a new and completely different one. I'm not familiar with healthcare IT systems, but telecommunications vendors don't typically build their systems to seamlessly work with two completely different business support systems at the same time, so some period of outage is unfortunately unavoidable.

 

 

 

drajk:

 

If all of the operational systems recording usage, maintaining balances to allow/disallow usage etc are still functioning correctly as is alleged then it would take a lot to convince me that the customer facing systems/reporting need to be down for a continuous one week period - to me the plan appears to be non customer-focused and suggestive that the upgrade is being done in a more cost effective way for the company rather than one which prioritises uptime and customer service.

 

 

 

 

At 2degrees, all of our customers are 100% real-time controlled. The system that measures and rates the usage and controls the network elements is not being replaced by this project. We will be upgrading some of its service logic during a very narrow window at the very end of the advertised period mainly so as to tweak the CDRs for the new billing system, but also to change the way notifications are handled, and to move from an old provisioning interface to a newer, more flexible one. As the developer of that service logic I have built a lot of safety valves into it to manage the transition so that there will be zero - not even an hour - planned interruption in its service availability to our customers.

 

However, billing and provisioning systems are a different kettle of fish. Having done basically this exact same project once before at a previous telco (not to mention that mobile is a very small industry and we all know the war stories about each other's upgrades, even the ones we weren't directly involved in) I assure you that there's no way around some outage. But also that this time round it's looking to be much quicker than the last one I worked on was.

 

We do apologise for the inconvenience.

 

 

Thanks salty, I appreciate the magnitude of what you are doing. I also accept that "some period of outage is unavoidable" however I find it hard to believe that this should be 168+ hours. If the new platform has been thoroughly tested pre-migration and there are efficient methods of data transfer available between the two systems then that is a very long time even with a million customers and the complexity of different customer types/plans/packages. If the usage/control data is not affected why couldn't the 'old' online account viewing be updated at least daily during such a long outage with clear disclaimer that the data may be up to 24 hours old. I also think it is poor that the methods left available to prepay customers to check balances do not provide detail of credit expiry dates (happy to be proven wrong re this but I can't see how one obtains this information).

 

In any case everybody is entitled to their own opinion - I just find it surprising that such a long outage is considered acceptable.


10 posts

Wannabe Geek
+1 received by user: 5


  # 1722684 19-Feb-2017 14:58
One person supports this post
Send private message

I totally agree with D. 

 

My background is also IT, including many years managing a well known AU Telcos Nonstop systems - this length of portal outage would not have been tolerated. 

 

If 2degrees feels it is imperative to perform a data migration which has a known lengthy impact to customer services, then it would be common courtesy to send out a couple of SMS's during the preceding weeks, warning of the upcoming customer portal outage and advising to topup via the webpage if required.   

 

 

 

 


4517 posts

Uber Geek
+1 received by user: 2623

Trusted
Lifetime subscriber

  # 1722701 19-Feb-2017 16:04
Send private message

Title of thread needs the word ' planned ' added

Linux

Stu

Hammered
5203 posts

Uber Geek
+1 received by user: 1155

Moderator
Trusted
Lifetime subscriber

  # 1722709 19-Feb-2017 16:20
Send private message

So will the automatic monthly pre-pay top-ups from a credit card happen? If not, will value packs roll over anyway to keep services working? What contingencies are in place to keep customers working if the upgrade falls over and top-ups don't go through? Will service be lost if you don't top-up with an alternative method?

 

Seems to me that with the systems that will be down, that 2Degrees really should have emailed and sent an SMS to every customer (or at least every customer likely to be affected).





Keep calm, and carry on posting.

 

 

 

Click to see full size Click to see full size


Stu

Hammered
5203 posts

Uber Geek
+1 received by user: 1155

Moderator
Trusted
Lifetime subscriber

  # 1722712 19-Feb-2017 16:25
One person supports this post
Send private message

Linux: Title of thread needs the word ' planned ' added

Linux

 

Done, although the fact it was planned should call for customers to be proactively advised in advance, and not just an entry on the Service status page!





Keep calm, and carry on posting.

 

 

 

Click to see full size Click to see full size


726 posts

Ultimate Geek
+1 received by user: 227

Trusted

  # 1722770 19-Feb-2017 19:06
Send private message

Agreed with above - I understand Telco systems are nasty, multi-headed beasts, but best practice for a known national outage would be some sort of notifications prior to the service outage period, not while it is in progress. It's not like 2degrees doesn't know how to get in touch with their customers.

2822 posts

Uber Geek
+1 received by user: 461

Trusted

  # 1722780 19-Feb-2017 19:44
Send private message

From memory didn't Spark have an outage for just about as long a while back?

 

If so then this is far from a unique situation


4 posts

Wannabe Geek
+1 received by user: 1


  # 1724353 22-Feb-2017 11:59
Send private message

I haven't been able to top up my phone balance for five days now! Why wasn't any warning given?

 

If 2DegreesMobile were actually sincere about customer inconvenience then they would have provided free or debt/IOU calling to prepaid customers whose balances expired and couldn't be topped up during the planned upgrade period, especially when they didn't provide any forewarning at all, just a technical web page notification which nobody would normally see.

 

What an incredibly arrogant lapse in customer service to not provide any forewarning via sms and media outlets when such a long period of disrupted service was planned. Sure, this may well have been a huge and necessary upgrade but its roll out has been a joke and the only conclusion that can be drawn from this experience this that 2Degreemobile clearly don't feel any sense of responsibility or goodwill towards their prepaid customers.


 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11
View this topic in a long page with up to 500 replies per page Create new topic



Twitter »

Follow us to receive Twitter updates when new discussions are posted in our forums:



Follow us to receive Twitter updates when news items and blogs are posted in our frontpage:



Follow us to receive Twitter updates when tech item prices are listed in our price comparison site:





News »

Porirua City Council first to adopt new council software solution Datascape
Posted 15-May-2019 12:00


New survey provides insight into schools' technology challenges and plans
Posted 15-May-2019 09:30


Apple Music now available on Alexa devices in Australia and New Zealand
Posted 15-May-2019 09:11


Make a stand against cyberbullying this Pink Shirt Day
Posted 14-May-2019 20:23


Samsung first TV manufacturer to launch the Apple TV App and Airplay 2
Posted 14-May-2019 20:11


Vodafone New Zealand sold
Posted 14-May-2019 07:25


Kordia boosts cloud performance with locally-hosted Microsoft Azure ExpressRoute
Posted 8-May-2019 10:25


Microsoft Azure ExpressRoute in New Zealand opens up faster, more secure internet for Kiwi businesses
Posted 8-May-2019 09:39


Vocus Communications to deliver Microsoft Azure Cloud Solutions through Azure ExpressRoute
Posted 8-May-2019 09:25


Independent NZ feature film #statusPending to premiere during WLG-X
Posted 6-May-2019 22:13


The ultimate dog photoshoot with Nokia 9 PureView #ForgottenDogsofInstagram
Posted 6-May-2019 09:41


Nokia 9 PureView available in New Zealand
Posted 6-May-2019 09:06


Motorola Solutions joins local partners to deliver advanced communications network in New Zealand
Posted 30-Apr-2019 21:50


Micron launches high-performance NVMe SSDs for cloud and enterprise markets
Posted 30-Apr-2019 10:27


Jaguar Land Rover trials in-vehicle smart wallet technology
Posted 29-Apr-2019 21:48



Geekzone Live »

Try automatic live updates from Geekzone directly in your browser, without refreshing the page, with Geekzone Live now.


Support Geekzone »

Our community of supporters help make Geekzone possible. Click the button below to join them.

Support Geezone on PressPatron



Are you subscribed to our RSS feed? You can download the latest headlines and summaries from our stories directly to your computer or smartphone by using a feed reader.

Alternatively, you can receive a daily email with Geekzone updates.