Texts sent to wrong phones...

Forums › 2degrees (including Slingshot, Orcon, Flip, Stuff Fibre, MyRepublic, 2talk and Vocus) › Texts sent to wrong phones...

1 | 2 | 3

3892 posts

Uber Geek

#496559 22-Jul-2011 10:17

johnr: Prod is not 100% the same as Pre prod, It's very very very close

Plans are always in place for rollback and this is why changes are deployed when most people are asleep,

John I do think we have to give Salt and his team credit for their roll back plans. One hour of disruption for something that I'm sure got any amount of pre testing, to be rolled back out, that's not bad in my book.

Clearly the pre prod system wasn't close enough on this occasion.

I don't know if you picked up the point of my last post.

'close' isn't good enough for a fault free production system. Fault free requires 1 to 1 replication, and even then that's some times not enough. Some times you need even more replication and resource in the test system than the production system so that you can not only run a test but also monitor what's going on.

However what I'm suggesting, is it realistic in a New Zealand network? Are New Zealanders willing to pay what it would cost to have fully replicated test systems?

My 200k phones in a warehouse was just a silly suggestion, but even the resource to set up 5,000 phones in such a way that you can do real world testing is really expensive and that cost has to be passed back to consumers.

My observation is that consumers simply aren't willing to pay for that resource on any of the New Zealand networks. I could be wrong.

In the past I've asked providers why they don't give users a heads up that a system test is going to be performed 'tonight'.

But even that suggestion isn't realistic is it? If you sent out such a text, how many people would hit the call center to object or ask if they can be excluded etc? Who pays for those calls to be answered? Consumers again don't want to pay.

I wonder if there should be a regulated test window? So each provider is required to do testing at the same time each week and those times are published? Each network would be staggered, so people wanting continuity of service could avoid outages.

The reason I propose regulation is to drive consumer awareness and ensure there is a fair and level playing field for all providers.

I also wonder if there should be a regulated opt in alerts system? Personally I'd be inclined to subscribe and then shut my phone off during the event window. Others may choose to have more than one provider and switch providers for that time frame, some just won't care and will just go back to sleep if their phone goes a bit bonkers for an hour.

If this was done, then when someone such as SteveOn gets up set, then we could at least point them to the outages notices and ask what ownership they took? At present Steve isn't really empowered to do anything.

Promote New Zealand - Get yourself a .kiwi.nz domain name!!!

Check out mine - i.am.a.can.do.kiwi.nz - don@i.am.a.can.do.kiwi.nz

timmmay

20578 posts

Uber Geek

Trusted

Lifetime subscriber

#496563 22-Jul-2011 10:23

Sure, they made a mistake, but they did it at a time that caused limited disruption and recovered quickly. Software's complex. It's almost impossible to have a perfect testing environment, even with general IT systems, I suspect with that hardware involved it's even more difficult.

I turn my cell off at night so it can't bother me, but I didn't have any problems during this event.

johnr

19282 posts

Uber Geek
Inactive user

#496564 22-Jul-2011 10:24

(Vodafone hat off here)

Don your post is a load of drible sorry, Regulated test window and other rubbish

Do you know how often upgrades , changes happen on a mobile network?

John

DonGould

3892 posts

Uber Geek

#496570 22-Jul-2011 10:29

timmmay: Sure, they made a mistake, but they did it at a time that caused limited disruption and recovered quickly. Software's complex. It's almost impossible to have a perfect testing environment, even with general IT systems, I suspect with that hardware involved it's even more difficult.

Yes. I agree with all that.

timmmay:
I turn my cell off at night so it can't bother me, but I didn't have any problems during this event.

How do I contact you in the case of emergency?

Promote New Zealand - Get yourself a .kiwi.nz domain name!!!

Check out mine - i.am.a.can.do.kiwi.nz - don@i.am.a.can.do.kiwi.nz

timmmay

20578 posts

Uber Geek

Trusted

Lifetime subscriber

#496574 22-Jul-2011 10:34

DonGould:
timmmay:
I turn my cell off at night so it can't bother me, but I didn't have any problems during this event.

How do I contact you in the case of emergency?

You can't. The cell's off, and I have no home phone. You come to my house, or call the police to come wake me up. When i'm sleeping, the most important thing to me is that i'm sleeping.

One day i'll get around to putting in a VoIP phone, i'd have done it already except for the couple of hundred dollars worth of hardware I need.

PenultimateHop

637 posts

Ultimate Geek

Trusted

#496580 22-Jul-2011 10:59

DonGould: However what I'm suggesting, is it realistic in a New Zealand network? Are New Zealanders willing to pay what it would cost to have fully replicated test systems?

My 200k phones in a warehouse was just a silly suggestion, but even the resource to set up 5,000 phones in such a way that you can do real world testing is really expensive and that cost has to be passed back to consumers.

My observation is that consumers simply aren't willing to pay for that resource on any of the New Zealand networks. I could be wrong.

In general they aren't. Witness both the price of products and the relatively slow cycle of products out of major telcos vs. smaller players and you can see which invests in quality and why prices are higher.

In general telcos are starting to reduce in-house testing windows due to the complexity and time it takes, and "trusting the vendor" more... Which is ironic given the previous nature of telcos and ISPs and their relative quality assurance requirements.

DonGould: I wonder if there should be a regulated test window? So each provider is required to do testing at the same time each week and those times are published? Each network would be staggered, so people wanting continuity of service could avoid outages.

This really isn't viable given the need for emergency fixes in networks.

DonGould

3892 posts

Uber Geek

#496582 22-Jul-2011 11:04

johnr: Don your post is a load of drible sorry, Regulated test window and other rubbish

Perhaps you'd like to explain why.

johnr: Do you know how often upgrades , changes happen on a mobile network?

No I don't. Perhaps you and Salty would like to educate us all on just how many change requests are pushed though daily.

I'm sure there would be many readers here who would be interested to know just how much work does go on behind the scenes each day that we simply take for granted.

Promote New Zealand - Get yourself a .kiwi.nz domain name!!!

Check out mine - i.am.a.can.do.kiwi.nz - don@i.am.a.can.do.kiwi.nz

Sharesies kids

Free kids accounts - trade shares and funds (NZ, US) with Sharesies (affiliate link).

DonGould

3892 posts

Uber Geek

#496589 22-Jul-2011 11:13

PenultimateHop:
In general telcos are starting to reduce in-house testing windows due to the complexity and time it takes,

Interesting.

PenultimateHop: and "trusting the vendor" more... Which is ironic given the previous nature of telcos and ISPs and their relative quality assurance requirements.

Now there's a scary concept... trusting the vendor... quick look at Telstra.au over the past decade for comment on how well that works out ;)

PenultimateHop: This really isn't viable given the need for emergency fixes in networks.

Why? Didn't Microsoft adopted 'fix Tuesday' (or is it Wednesday) after public pressure?

I get the whole emergency thing. That is obvious. And I get that the minute you write a regulation that says 'Monday only... or emergency' then suddenly everything might become an emergency.

But currently we've got 'what ever, when ever...' and none of us using this technology can do any management, but telcos can still stuff up - as is seen here.

Observation - SteveOn gets really upset, to the point where he posts simply offensive comments, I ask about regulation and get rubbished, he gets banned for a month.

At the same time our government is going on about building open access and transparent systems.

humm... am I just joining wrong dots today?

Promote New Zealand - Get yourself a .kiwi.nz domain name!!!

Check out mine - i.am.a.can.do.kiwi.nz - don@i.am.a.can.do.kiwi.nz

freitasm

BDFL - Memuneh
79263 posts

Uber Geek

Administrator

ID Verified

Trusted

Geekzone

Lifetime subscriber

#496599 22-Jul-2011 11:29

He got suspended because of the language used, not because of the question.

PenultimateHop

637 posts

Ultimate Geek

Trusted

#496601 22-Jul-2011 11:30

DonGould: Now there's a scary concept... trusting the vendor... quick look at Telstra.au over the past decade for comment on how well that works out ;)

It's probably safe to assume there are significant penalties attached to failure and that the vendors do try their best. However emergency and critical fixes are always risky.

DonGould:
Why? Didn't Microsoft adopted 'fix Tuesday' (or is it Wednesday) after public pressure?

Patch Tuesday exists to allow corporate system administrators to plan for it more than anything else. Microsoft still releases regular updates outside of Patch Tuesday and has released out-of-band urgent fixes as well - it would be irresponsible not to (and Patch Tuesday has of course lead to Exploit Wednesday).

DonGould:
I get the whole emergency thing. That is obvious. And I get that the minute you write a regulation that says 'Monday only... or emergency' then suddenly everything might become an emergency.

Declaring everything as an emergency is one way to handle it (and I've had to do that in the past to get around inflexible change management practices at one organization).

DonGould:
But currently we've got 'what ever, when ever...' and none of us using this technology can do any management, but telcos can still stuff up - as is seen here.

The alternative is to have a platform running in a risky environment. The fact that it was a binary patch says something interesting.

Would you rather the operators proceeded with cautious speed to ensure the system stays working; or allowed it to risk complete failure while waiting for "Reboot Monday"?

It annoys me that banks have their nightly processing freeze (often at random times) and randomly take their Internet Banking services offline without warning, but I accept they have to do that to keep it secure or reconcile my cents. It's reasonable and expected that network operators do the same thing.

DonGould: At the same time our government is going on about building open access and transparent systems.

humm... am I just joining wrong dots today?

I've no idea what you're implying.

freitasm

BDFL - Memuneh
79263 posts

Uber Geek

Administrator

ID Verified

Trusted

Geekzone

Lifetime subscriber

#496602 22-Jul-2011 11:31

As for Telco releases, what a difference from the 80s and 90s (when I started working in Telco space)... That's what happens when techonology development doesn't keep up with demand.

jaymz

1133 posts

Uber Geek

#496609 22-Jul-2011 11:39

In an ideal world, all systems would have a test enviroment for testing and confirming updates will not incur any issues. but unfoturnately in some cases this is not possible.

Surely, those who work in IT will understand the complications and costs involved with running test systems:
There is the cost of aditional hardware (does the client cover this or the IT support?)
The cost of taking time out of a users day to test the change
The time that the IT support spend updating the test system (effectivley doubling the time for the change)

In my experience, running a test system only serves to prove the process itself will work, not if it will fix the issue.

IMO a robust fall back proceedure is a far better use of time and resources.

Of course this doesnt apply to every problem/fix, if you can replicate the error easily then a test system is a good option.

On the topic that SteveON mentioned ("Important txt's lost"), people assume that SMS and EMail systems are full proof and guarenteed. This is simply not the case, you will find there will be no one who guarantees their email system will deliver email everytime. There are simply too many external factors that can go wrong that no one company can guard against.
It goes without saying (or so i thought) if a message is so important, then you should follow up with a phone call or something to confirm the other person had recieved the message.

My 2 cents on the whole thing anyway.

DonGould

3892 posts

Uber Geek

#496624 22-Jul-2011 11:58

jaymz:
On the topic that SteveON mentioned ("Important txt's lost"), people assume that SMS and EMail systems are full proof and guarenteed. This is simply not the case, you will find there will be no one who guarantees their email system will deliver email everytime.

Both protocols have systems in them to ensure message delivery. They're just not used.

Last time I asked about sms, iirc I was told that the message confirmation aspect of the protocol is not used in .nz because of the cost.

Promote New Zealand - Get yourself a .kiwi.nz domain name!!!

Check out mine - i.am.a.can.do.kiwi.nz - don@i.am.a.can.do.kiwi.nz

jaymz

1133 posts

Uber Geek

#496631 22-Jul-2011 12:21

DonGould:
Both protocols have systems in them to ensure message delivery. They're just not used.

Last time I asked about sms, iirc I was told that the message confirmation aspect of the protocol is not used in .nz because of the cost.

Exactly, until those confirmation systems are "enforced" then people should employ their own confirmation process (follow up phone call).

The downside of the confirmation systems is that they use the same technology that the original message was sent via. We have all seen emails reach their destination but not come back (issue with receiver's mail system)

DonGould

3892 posts

Uber Geek

#496633 22-Jul-2011 12:26

jaymz:Exactly, until those confirmation systems are "enforced" then people should employ their own confirmation process (follow up phone call).

Currently I can't even buy them on the mobile network as far as I'm aware.

With email, we can both agree to use message confirmation and read protocols and agree to set up a network space that means normal spam systems won't wreck out confirmation messaging.

Promote New Zealand - Get yourself a .kiwi.nz domain name!!!

Check out mine - i.am.a.can.do.kiwi.nz - don@i.am.a.can.do.kiwi.nz

1 | 2 | 3