Geekzone: technology news, blogs, forums
Guest
Welcome Guest.
You haven't logged in yet. If you don't have an account you can register now.


nzkiwiman

2585 posts

Uber Geek
+1 received by user: 390

Subscriber

#208356 8-Feb-2017 09:47
Send private message

I work as part of a 8 man IT team on the helpdesk (but with a fancy title) and over the last few years have become increasingly frustrated and annoyed at our SysOps who constantly make changes to the environment outside of the designated weekend and do not communicate that these changes have been made. As a result, when things go wrong, the first I know about the changes is due to the calls from our users.

 

I have spoken to management about this lack of communication from the team and it may "resolve" for a few months before going back to the way it was, which then gets me fired up again which leads to another talk to management which leads to a few months of communication before the cycle starts again.

 

 

 

As a result, I am wondering how other IT teams manage internal communication of changes?
Or are we all dysfunctional and due to our industry none of us can communicate with each other?

 

 

 

{hopefully I have remained vague enough not to be identified should my colleagues also use Geekzone, but too bad if they do}


Filter this topic showing only the reply marked as answer View this topic in a long page with up to 500 replies per page Create new topic
 1 | 2

xpd

xpd
Geek of Coastguard
14115 posts

Uber Geek
+1 received by user: 4574

Retired Mod
ID Verified
Trusted
Lifetime subscriber

  #1717080 8-Feb-2017 10:13
Send private message

Its a common issue.

 

Had it at most of my jobs, some small teams, some large multi-nationals, and always got left out of comms regarding user/customer systems being changed/updated etc.

 

ITIL might help towards it, but just comes down to someone being responsible for relaying information.





XPD / Gavin

 

LinkTree

 

 

 




kiwi_64
276 posts

Ultimate Geek
+1 received by user: 142

Subscriber

  #1717082 8-Feb-2017 10:14
Send private message

Does your organisation use standard/accepted processes for IT Management/Change Control?  If so, I assume all changes are being cycled through Change Review processes.  Why not have the Service Desk leader included as a mandatory approver to all changes?  Even if they can't provide technical input, it does mean that they will/must be aware of all changes and dates prior to implementation,


deadlyllama
1283 posts

Uber Geek
+1 received by user: 476

Trusted

  #1717124 8-Feb-2017 11:26
Send private message

Has your company grown from a small outfit where there were only a couple of staff who talked to each other anyway, to something bigger?  Are the SysOps staff from when the company was smaller?  If so, this sounds familiar...

 

It sounds like you're making this your problem when it needs to become the SysOps staff's problem.  Is there a way they can share your pain?  e.g. DR drills which fail because the infrastructure documentation wasn't updated to reflect the current state of the environment, on call staff who have to wake up other staff because something is broken and the documentation is out of date...




StarBlazer
961 posts

Ultimate Geek
+1 received by user: 218

Trusted

  #1717184 8-Feb-2017 13:17
Send private message

Unless it's a large organisation with robust change control processes, what you are experiencing is common.

 

At a very low level, a lot of teams have a team mailbox or distribution group - ask to be added to this so that you can monitor impending changes.  

 

At a higher level a monthly meeting of team leads where infrastructure changes are discussed - but the reality is decisions are made and changes implemented between meeting cycles.

 

As mentioned earlier it is hard to change small business mentality as the company grows.





Procrastination eventually pays off.


ubergeeknz
3344 posts

Uber Geek
+1 received by user: 1041

Trusted
Vocus

  #1717194 8-Feb-2017 13:34
Send private message

The only real solution to ensuring notification AND improving change outcomes is a proper change control process, with peer review, a change approval board, etc.

 

But a good start would be to have the SysOps notify a small mailing list (perhaps, only covering the IT department / helpdesk) when they're about to make a change and when they're done, outlining what the change entails.  This at least creates some level of transparency and accountability.

 

However such a policy (as with any kind of change control) needs to be strictly enforced - or it's a waste of time.


gzt

gzt
18674 posts

Uber Geek
+1 received by user: 7809

Lifetime subscriber

  #1717196 8-Feb-2017 13:36
Send private message

nzkiwiman:

I work as part of a 8 man IT team on the helpdesk (but with a fancy title) and over the last few years have become increasingly frustrated and annoyed at our SysOps who constantly make changes to the environment outside of the designated weekend and do not communicate that these changes have been made. As a result, when things go wrong, the first I know about the changes is due to the calls from our users.


Depends on the nature of the changes. Are these admin scripts and under source control? If not, maybe they should be. Then you have change notification on those scripts.

 
 
 

Shop now at Mighty Ape (affiliate link).

gzt

gzt
18674 posts

Uber Geek
+1 received by user: 7809

Lifetime subscriber

  #1717201 8-Feb-2017 13:51
Send private message

There may be another side to the story. From their perspective they probably make lots of changes and very few of those have any user impact.

antoniosk
2382 posts

Uber Geek
+1 received by user: 742

ID Verified
Trusted
Lifetime subscriber

  #1717212 8-Feb-2017 14:07
Send private message

In a corporate it is not uncommon for the affected depts to spend some time sizing the impact of outage and hitting the IT budget for compensation.

It's petty and childish, but after a few $20k hits people started paying attention. At a now defunct telco it was not uncommon for the ops team to drive changes that caused switching breaks - leading to outages for some customers lasting several hours because the downstream systems could not accommodate. Queue reps for comp.

The technology of the day was not resilient, but the principle was that nearly all the changes were for convenience rather than emergency.




________

 

Antoniosk


nzkiwiman

2585 posts

Uber Geek
+1 received by user: 390

Subscriber

  #1717218 8-Feb-2017 14:24
Send private message

Apparently we are running under ITIL, though from working here I know that is a complete farce/joke.

 

While the entire SysOps team is bad at communication, the longest serving member of the team is the most common offender. Sadly he is also in charge of the change control process and makes life very hard for me if my documentation for changes is not "up to scratch" but does not follow the process himself (as with the most recent case over the long weekend).

 

Over the weekend we had a major change in the back end policy that controls our Citrix environment, used by 95% of the staff in the company. 
There was no communication from the SysOps who put that change in place, and in fact did not even tell his colleagues he had made the change. There was no change put into our change control process, and we are several weeks away from the next scheduled change weekend. 
The change was not tested fully and as a result users called for support, making me very angry when I figured out what had changed.

 

We have all the email systems setup so it takes 15 seconds to notify everyone in the team that a change has been made ...


doublek69
36 posts

Geek
+1 received by user: 16


  #1717240 8-Feb-2017 15:33
Send private message

Pathway 1 = have the CIO/Head of IT/Whoever holds the budget for IT, mandate that for the next unapproved change, a termination of said employee will result. The Nuclear Option...

 

Pathway 2 = find a friendly SysOp, buy him/her coffee and formulate a relationship where they give you a heads-up on changes. The Comrade Option...

 

Pathway 3 = disable the Change Manager's account next time it happens. Don't re-enable it until they agree to tell you about changes. The Hostage Option...

 

 

 

Good Luck!

 

 


lxsw20
3689 posts

Uber Geek
+1 received by user: 2174

Subscriber

#1717429 8-Feb-2017 23:03
Send private message

gzt: There may be another side to the story. From their perspective they probably make lots of changes and very few of those have any user impact.

 

 

 

Some times user scream testing = best testing.


 
 
 

Move to New Zealand's best fibre broadband service (affiliate link). Free setup code: R587125ERQ6VE. Note that to use Quic Broadband you must be comfortable with configuring your own router.
Peamsable
98 posts

Master Geek
+1 received by user: 55
Inactive user


  #1717460 9-Feb-2017 07:32

Forward the help desk phone to his mobile.

nzkiwiman

2585 posts

Uber Geek
+1 received by user: 390

Subscriber

  #1717463 9-Feb-2017 07:41
Send private message

doublek69:

 

Pathway 1 = have the CIO/Head of IT/Whoever holds the budget for IT, mandate that for the next unapproved change, a termination of said employee will result. The Nuclear Option...

 

Pathway 2 = find a friendly SysOp, buy him/her coffee and formulate a relationship where they give you a heads-up on changes. The Comrade Option...

 

Pathway 3 = disable the Change Manager's account next time it happens. Don't re-enable it until they agree to tell you about changes. The Hostage Option...

 

 

 

Good Luck!

 

 

 

 

 

 

How I wish any of these were an option

 

Option 1 - he is "best buds" with my manager, his manager and the big boss and he will never be fired (holds too many other cards close to his chest)

 

Option 2 - the others are just as bad and in the most recent weekend example, they didn't know about the change either

 

Option 3 - more than likely to get me fired

 

 

 

spencer: Forward the help desk phone to his mobile.

 

This probably my only option; check emails after change weekend - if no communication then forward phones (if communication, do job)


deadlyllama
1283 posts

Uber Geek
+1 received by user: 476

Trusted

  #1717502 9-Feb-2017 09:32
Send private message

My unhelpful advice on dealing with problems at work (and this is what I've done in the past):

 

1. Talk with your manager.

 

2. Wait for something to happen.

 

3. If nothing happens, talk with your manager.

 

4. Wait for something to happen.

 

5. Give up waiting (after 6-12 months), and resign.  Citing the problem if your manager isn't a dick who will give you a bad reference.

 

It sounds like you're awfully close to step 5.  It's not one I've taken without having another job lined up.

 

I had to leave a job that was fun and technically challenging, because it caused me far too much stress.  I'd get given urgent problems to fix and I'd move heaven and earth to fix them.  Got sick of the relentless set of urgent issues and my manager said he'd try assigning them to other people.  He did that, the other people would usually give up after a week and I'd have to fix them anyway, now a week more urgent than they were before.


michaelmurfy
meow
13579 posts

Uber Geek
+1 received by user: 10910

Moderator
ID Verified
Trusted
Lifetime subscriber

  #1717527 9-Feb-2017 10:19
Send private message

How about creating a slack account for the company and asking the person nicely for the benefit of everyone if he puts planned / implemented changes in there? There isn't any need for either office poop chucking or a hostage situation, just explain the business case (less wasted time etc) and be nice about it.

As a DevOPS engineer in my previous roles I always HATED having change management policies but I do see the point of it. At one point one of my peers decided it was a great idea to deviate from the standard system build (Debian based, postgresql) to a new standard of CentOS, MySQL and a whole bunch of security rules. Needless to say this broke many things and the devs got pretty upset (understandable) - when I spoke out and tried to fight it I ended up getting made redundant then asked back 3mo later to help clean up the mess (of which, I declined).




Michael Murphy | https://murfy.nz
Referral Links: Quic Broadband (use R122101E7CV7Q for free setup)

Are you happy with what you get from Geekzone? Please consider supporting us by subscribing.
Opinions are my own and not the views of my employer.


 1 | 2
Filter this topic showing only the reply marked as answer View this topic in a long page with up to 500 replies per page Create new topic








Geekzone Live »

Try automatic live updates from Geekzone directly in your browser, without refreshing the page, with Geekzone Live now.



Are you subscribed to our RSS feed? You can download the latest headlines and summaries from our stories directly to your computer or smartphone by using a feed reader.