Geekzone: technology news, blogs, forums
Guest
Welcome Guest.
You haven't logged in yet. If you don't have an account you can register now.
View this topic in a long page with up to 500 replies per page Create new topic
1 | 2 | 3
3889 posts

Uber Geek
+1 received by user: 163


  Reply # 496559 22-Jul-2011 10:17
Send private message

johnr: Prod is not 100% the same as Pre prod, It's very very very close

Plans are always in place for rollback and this is why changes are deployed when most people are asleep,



John I do think we have to give Salt and his team credit for their roll back plans.  One hour of disruption for something that I'm sure got any amount of pre testing, to be rolled back out, that's not bad in my book.

Clearly the pre prod system wasn't close enough on this occasion.

I don't know if you picked up the point of my last post.

'close' isn't good enough for a fault free production system.  Fault free requires 1 to 1 replication, and even then that's some times not enough.  Some times you need even more replication and resource in the test system than the production system so that you can not only run a test but also monitor what's going on.

However what I'm suggesting, is it realistic in a New Zealand network?  Are New Zealanders willing to pay what it would cost to have fully replicated test systems? 

My 200k phones in a warehouse was just a silly suggestion, but even the resource to set up 5,000 phones in such a way that you can do real world testing is really expensive and that cost has to be passed back to consumers.

My observation is that consumers simply aren't willing to pay for that resource on any of the New Zealand networks.  I could be wrong.

In the past I've asked providers why they don't give users a heads up that a system test is going to be performed 'tonight'.

But even that suggestion isn't realistic is it?  If you sent out such a text, how many people would hit the call center to object or ask if they can be excluded etc?  Who pays for those calls to be answered?  Consumers again don't want to pay.

I wonder if there should be a regulated test window?  So each provider is required to do testing at the same time each week and those times are published?  Each network would be staggered, so people wanting continuity of service could avoid outages.

The reason I propose regulation is to drive consumer awareness and ensure there is a fair and level playing field for all providers.

I also wonder if there should be a regulated opt in alerts system?   Personally I'd be inclined to subscribe and then shut my phone off during the event window.  Others may choose to have more than one provider and switch providers for that time frame, some just won't care and will just go back to sleep if their phone goes a bit bonkers for an hour.

If this was done, then when someone such as SteveOn gets up set, then we could at least point them to the outages notices and ask what ownership they took?  At present Steve isn't really empowered to do anything.






Promote New Zealand - Get yourself a .kiwi.nz domain name!!!

Check out mine - i.am.a.can.do.kiwi.nz - don@i.am.a.can.do.kiwi.nz


14151 posts

Uber Geek
+1 received by user: 2546

Trusted
Subscriber

  Reply # 496563 22-Jul-2011 10:23
Send private message

Sure, they made a mistake, but they did it at a time that caused limited disruption and recovered quickly. Software's complex. It's almost impossible to have a perfect testing environment, even with general IT systems, I suspect with that hardware involved it's even more difficult.

I turn my cell off at night so it can't bother me, but I didn't have any problems during this event.




AWS Certified Solution Architect Professional, Sysop Administrator Associate, and Developer Associate
TOGAF certified enterprise architect
Professional photographer


19282 posts

Uber Geek
+1 received by user: 2600
Inactive user


  Reply # 496564 22-Jul-2011 10:24
Send private message

(Vodafone hat off here)

Don your post is a load of drible sorry, Regulated test window and other rubbish


Do you know how often upgrades , changes happen on a mobile network?

John

3889 posts

Uber Geek
+1 received by user: 163


  Reply # 496570 22-Jul-2011 10:29
Send private message

timmmay: Sure, they made a mistake, but they did it at a time that caused limited disruption and recovered quickly. Software's complex. It's almost impossible to have a perfect testing environment, even with general IT systems, I suspect with that hardware involved it's even more difficult.


Yes.  I agree with all that.

timmmay:
I turn my cell off at night so it can't bother me, but I didn't have any problems during this event.


How do I contact you in the case of emergency?






Promote New Zealand - Get yourself a .kiwi.nz domain name!!!

Check out mine - i.am.a.can.do.kiwi.nz - don@i.am.a.can.do.kiwi.nz


14151 posts

Uber Geek
+1 received by user: 2546

Trusted
Subscriber

  Reply # 496574 22-Jul-2011 10:34
Send private message

DonGould:
timmmay:
I turn my cell off at night so it can't bother me, but I didn't have any problems during this event.


How do I contact you in the case of emergency?


You can't. The cell's off, and I have no home phone. You come to my house, or call the police to come wake me up. When i'm sleeping, the most important thing to me is that i'm sleeping.

One day i'll get around to putting in a VoIP phone, i'd have done it already except for the couple of hundred dollars worth of hardware I need.





AWS Certified Solution Architect Professional, Sysop Administrator Associate, and Developer Associate
TOGAF certified enterprise architect
Professional photographer


637 posts

Ultimate Geek
+1 received by user: 2

Trusted

  Reply # 496580 22-Jul-2011 10:59
Send private message

DonGould: However what I'm suggesting, is it realistic in a New Zealand network?  Are New Zealanders willing to pay what it would cost to have fully replicated test systems? 

My 200k phones in a warehouse was just a silly suggestion, but even the resource to set up 5,000 phones in such a way that you can do real world testing is really expensive and that cost has to be passed back to consumers.

My observation is that consumers simply aren't willing to pay for that resource on any of the New Zealand networks.  I could be wrong.

In general they aren't.  Witness both the price of products and the relatively slow cycle of products out of major telcos vs. smaller players and you can see which invests in quality and why prices are higher.

In general telcos are starting to reduce in-house testing windows due to the complexity and time it takes, and "trusting the vendor" more...  Which is ironic given the previous nature of telcos and ISPs and their relative quality assurance requirements.

DonGould: I wonder if there should be a regulated test window?  So each provider is required to do testing at the same time each week and those times are published?  Each network would be staggered, so people wanting continuity of service could avoid outages.

This really isn't viable given the need for emergency fixes in networks.

3889 posts

Uber Geek
+1 received by user: 163


  Reply # 496582 22-Jul-2011 11:04
Send private message

johnr: Don your post is a load of drible sorry, Regulated test window and other rubbish


Perhaps you'd like to explain why.


johnr: Do you know how often upgrades , changes happen on a mobile network?


No I don't.  Perhaps you and Salty would like to educate us all on just how many change requests are pushed though daily. 

I'm sure there would be many readers here who would be interested to know just how much work does go on behind the scenes each day that we simply take for granted.








Promote New Zealand - Get yourself a .kiwi.nz domain name!!!

Check out mine - i.am.a.can.do.kiwi.nz - don@i.am.a.can.do.kiwi.nz


3889 posts

Uber Geek
+1 received by user: 163


  Reply # 496589 22-Jul-2011 11:13
Send private message

PenultimateHop:
In general telcos are starting to reduce in-house testing windows due to the complexity and time it takes,


Interesting.

PenultimateHop: and "trusting the vendor" more...  Which is ironic given the previous nature of telcos and ISPs and their relative quality assurance requirements.



Now there's a scary concept... trusting the vendor...  quick look at Telstra.au over the past decade for comment on how well that works out ;)

PenultimateHop: This really isn't viable given the need for emergency fixes in networks.


Why?  Didn't Microsoft adopted 'fix Tuesday' (or is it Wednesday) after public pressure?

I get the whole emergency thing.  That is obvious.  And I get that the minute you write a regulation that says 'Monday only... or emergency' then suddenly everything might become an emergency.

But currently we've got 'what ever, when ever...' and none of us using this technology can do any management, but telcos can still stuff up - as is seen here. 

Observation - SteveOn gets really upset, to the point where he posts simply offensive comments, I ask about regulation and get rubbished, he gets banned for a month.

At the same time our government is going on about building open access and transparent systems.

humm...  am I just joining wrong dots today?






Promote New Zealand - Get yourself a .kiwi.nz domain name!!!

Check out mine - i.am.a.can.do.kiwi.nz - don@i.am.a.can.do.kiwi.nz


BDFL - Memuneh
61336 posts

Uber Geek
+1 received by user: 12084

Administrator
Trusted
Geekzone
Lifetime subscriber

637 posts

Ultimate Geek
+1 received by user: 2

Trusted

  Reply # 496601 22-Jul-2011 11:30
Send private message

DonGould: Now there's a scary concept... trusting the vendor...  quick look at Telstra.au over the past decade for comment on how well that works out ;)

It's probably safe to assume there are significant penalties attached to failure and that the vendors do try their best. However emergency and critical fixes are always risky.
DonGould:
Why?  Didn't Microsoft adopted 'fix Tuesday' (or is it Wednesday) after public pressure?

Patch Tuesday exists to allow corporate system administrators to plan for it more than anything else.  Microsoft still releases regular updates outside of Patch Tuesday and has released out-of-band urgent fixes as well - it would be irresponsible not to (and Patch Tuesday has of course lead to Exploit Wednesday).
DonGould:
I get the whole emergency thing.  That is obvious.  And I get that the minute you write a regulation that says 'Monday only... or emergency' then suddenly everything might become an emergency.

Declaring everything as an emergency is one way to handle it (and I've had to do that in the past to get around inflexible change management practices at one organization).
DonGould:
But currently we've got 'what ever, when ever...' and none of us using this technology can do any management, but telcos can still stuff up - as is seen here.

The alternative is to have a platform running in a risky environment.  The fact that it was a binary patch says something interesting.

Would you rather the operators proceeded with cautious speed to ensure the system stays working; or allowed it to risk complete failure while waiting for "Reboot Monday"?

It annoys me that banks have their nightly processing freeze (often at random times) and randomly take their Internet Banking services offline without warning, but I accept they have to do that to keep it secure or reconcile my cents.  It's reasonable and expected that network operators do the same thing.

DonGould: At the same time our government is going on about building open access and transparent systems.

humm...  am I just joining wrong dots today?

I've no idea what you're implying.

BDFL - Memuneh
61336 posts

Uber Geek
+1 received by user: 12084

Administrator
Trusted
Geekzone
Lifetime subscriber

  Reply # 496602 22-Jul-2011 11:31
Send private message

As for Telco releases, what a difference from the 80s and 90s (when I started working in Telco space)... That's what happens when techonology development doesn't keep up with demand.




1086 posts

Uber Geek
+1 received by user: 65


  Reply # 496609 22-Jul-2011 11:39
Send private message

In an ideal world, all systems would have a test enviroment for testing and confirming updates will not incur any issues. but unfoturnately in some cases this is not possible.

Surely, those who work in IT will understand the complications and costs involved with running test systems:
There is the cost of aditional hardware (does the client cover this or the IT support?)
The cost of taking time out of a users day to test the change
The time that the IT support spend updating the test system (effectivley doubling the time for the change)

In my experience, running a test system only serves to prove the process itself will work, not if it will fix the issue.

IMO a robust fall back proceedure is a far better use of time and resources. 

Of course this doesnt apply to every problem/fix, if you can replicate the error easily then a test system is a good option.

On the topic that SteveON mentioned ("Important txt's lost"), people assume that SMS and EMail systems are full proof and guarenteed.  This is simply not the case, you will find there will be no one who guarantees their email system will deliver email everytime.  There are simply too many external factors that can go wrong that no one company can guard against.
It goes without saying (or so i thought) if a message is so important, then you should follow up with a phone call or something to confirm the other person had recieved the message.

My 2 cents on the whole thing anyway.

3889 posts

Uber Geek
+1 received by user: 163


  Reply # 496624 22-Jul-2011 11:58
Send private message

jaymz:
On the topic that SteveON mentioned ("Important txt's lost"), people assume that SMS and EMail systems are full proof and guarenteed.  This is simply not the case, you will find there will be no one who guarantees their email system will deliver email everytime.


Both protocols have systems in them to ensure message delivery.  They're just not used.

Last time I asked about sms, iirc I was told that the message confirmation aspect of the protocol is not used in .nz because of the cost.






Promote New Zealand - Get yourself a .kiwi.nz domain name!!!

Check out mine - i.am.a.can.do.kiwi.nz - don@i.am.a.can.do.kiwi.nz


1086 posts

Uber Geek
+1 received by user: 65


  Reply # 496631 22-Jul-2011 12:21
Send private message

DonGould:
Both protocols have systems in them to ensure message delivery.  They're just not used.

Last time I asked about sms, iirc I was told that the message confirmation aspect of the protocol is not used in .nz because of the cost.


Exactly, until those confirmation systems are "enforced" then people should employ their own confirmation process (follow up phone call).

The downside of the confirmation systems is that they use the same technology that the original message was sent via.  We have all seen emails reach their destination but not come back (issue with receiver's mail system)

3889 posts

Uber Geek
+1 received by user: 163


  Reply # 496633 22-Jul-2011 12:26
Send private message

jaymz:Exactly, until those confirmation systems are "enforced" then people should employ their own confirmation process (follow up phone call).


Currently I can't even buy them on the mobile network as far as I'm aware.

With email, we can both agree to use message confirmation and read protocols and agree to set up a network space that means normal spam systems won't wreck out confirmation messaging.






Promote New Zealand - Get yourself a .kiwi.nz domain name!!!

Check out mine - i.am.a.can.do.kiwi.nz - don@i.am.a.can.do.kiwi.nz


1 | 2 | 3
View this topic in a long page with up to 500 replies per page Create new topic

Twitter »

Follow us to receive Twitter updates when new discussions are posted in our forums:



Follow us to receive Twitter updates when news items and blogs are posted in our frontpage:



Follow us to receive Twitter updates when tech item prices are listed in our price comparison site:



Geekzone Live »

Try automatic live updates from Geekzone directly in your browser, without refreshing the page, with Geekzone Live now.



Are you subscribed to our RSS feed? You can download the latest headlines and summaries from our stories directly to your computer or smartphone by using a feed reader.

Alternatively, you can receive a daily email with Geekzone updates.