Geekzone: technology news, blogs, forums
Guest
Welcome Guest.
You haven't logged in yet. If you don't have an account you can register now.




2194 posts

Uber Geek
+1 received by user: 186


# 208356 8-Feb-2017 09:47
One person supports this post
Send private message

I work as part of a 8 man IT team on the helpdesk (but with a fancy title) and over the last few years have become increasingly frustrated and annoyed at our SysOps who constantly make changes to the environment outside of the designated weekend and do not communicate that these changes have been made. As a result, when things go wrong, the first I know about the changes is due to the calls from our users.

 

I have spoken to management about this lack of communication from the team and it may "resolve" for a few months before going back to the way it was, which then gets me fired up again which leads to another talk to management which leads to a few months of communication before the cycle starts again.

 

 

 

As a result, I am wondering how other IT teams manage internal communication of changes?
Or are we all dysfunctional and due to our industry none of us can communicate with each other?

 

 

 

{hopefully I have remained vague enough not to be identified should my colleagues also use Geekzone, but too bad if they do}


Filter this topic showing only the reply marked as answer View this topic in a long page with up to 500 replies per page Create new topic
 1 | 2

xpd

Chief Trash Bandit
9498 posts

Uber Geek
+1 received by user: 1615

Mod Emeritus
Trusted
Lifetime subscriber

  # 1717080 8-Feb-2017 10:13
One person supports this post
Send private message

Its a common issue.

 

Had it at most of my jobs, some small teams, some large multi-nationals, and always got left out of comms regarding user/customer systems being changed/updated etc.

 

ITIL might help towards it, but just comes down to someone being responsible for relaying information.





XPD / Gavin / DemiseNZ

 

Server : i3-3240 @ 3.40GHz  16GB RAM  Win 10 Pro    Workstation : i5-3570K @ 3.40GHz  16GB RAM  RX580 4GB Win 10 Pro    Console : Xbox One

 

https://www.xpd.co.nz - Games, geeks, and more.     Now on BigPipe 100/100 and 2Talk


176 posts

Master Geek
+1 received by user: 29


  # 1717082 8-Feb-2017 10:14
Send private message

Does your organisation use standard/accepted processes for IT Management/Change Control?  If so, I assume all changes are being cycled through Change Review processes.  Why not have the Service Desk leader included as a mandatory approver to all changes?  Even if they can't provide technical input, it does mean that they will/must be aware of all changes and dates prior to implementation,


 
 
 
 


907 posts

Ultimate Geek
+1 received by user: 287


  # 1717124 8-Feb-2017 11:26
Send private message

Has your company grown from a small outfit where there were only a couple of staff who talked to each other anyway, to something bigger?  Are the SysOps staff from when the company was smaller?  If so, this sounds familiar...

 

It sounds like you're making this your problem when it needs to become the SysOps staff's problem.  Is there a way they can share your pain?  e.g. DR drills which fail because the infrastructure documentation wasn't updated to reflect the current state of the environment, on call staff who have to wake up other staff because something is broken and the documentation is out of date...


961 posts

Ultimate Geek
+1 received by user: 224

Trusted
Subscriber

  # 1717184 8-Feb-2017 13:17
Send private message

Unless it's a large organisation with robust change control processes, what you are experiencing is common.

 

At a very low level, a lot of teams have a team mailbox or distribution group - ask to be added to this so that you can monitor impending changes.  

 

At a higher level a monthly meeting of team leads where infrastructure changes are discussed - but the reality is decisions are made and changes implemented between meeting cycles.

 

As mentioned earlier it is hard to change small business mentality as the company grows.





Procrastination eventually pays off.


3344 posts

Uber Geek
+1 received by user: 1089

Trusted
Vocus

  # 1717194 8-Feb-2017 13:34
Send private message

The only real solution to ensuring notification AND improving change outcomes is a proper change control process, with peer review, a change approval board, etc.

 

But a good start would be to have the SysOps notify a small mailing list (perhaps, only covering the IT department / helpdesk) when they're about to make a change and when they're done, outlining what the change entails.  This at least creates some level of transparency and accountability.

 

However such a policy (as with any kind of change control) needs to be strictly enforced - or it's a waste of time.


gzt

10705 posts

Uber Geek
+1 received by user: 1760


  # 1717196 8-Feb-2017 13:36
Send private message

nzkiwiman:

I work as part of a 8 man IT team on the helpdesk (but with a fancy title) and over the last few years have become increasingly frustrated and annoyed at our SysOps who constantly make changes to the environment outside of the designated weekend and do not communicate that these changes have been made. As a result, when things go wrong, the first I know about the changes is due to the calls from our users.


Depends on the nature of the changes. Are these admin scripts and under source control? If not, maybe they should be. Then you have change notification on those scripts.

gzt

10705 posts

Uber Geek
+1 received by user: 1760


  # 1717201 8-Feb-2017 13:51
Send private message

There may be another side to the story. From their perspective they probably make lots of changes and very few of those have any user impact.

1884 posts

Uber Geek
+1 received by user: 456

Trusted
Subscriber

  # 1717212 8-Feb-2017 14:07
Send private message

In a corporate it is not uncommon for the affected depts to spend some time sizing the impact of outage and hitting the IT budget for compensation.

It's petty and childish, but after a few $20k hits people started paying attention. At a now defunct telco it was not uncommon for the ops team to drive changes that caused switching breaks - leading to outages for some customers lasting several hours because the downstream systems could not accommodate. Queue reps for comp.

The technology of the day was not resilient, but the principle was that nearly all the changes were for convenience rather than emergency.




________

 

Antonios K

 

Click to see full size




2194 posts

Uber Geek
+1 received by user: 186


  # 1717218 8-Feb-2017 14:24
One person supports this post
Send private message

Apparently we are running under ITIL, though from working here I know that is a complete farce/joke.

 

While the entire SysOps team is bad at communication, the longest serving member of the team is the most common offender. Sadly he is also in charge of the change control process and makes life very hard for me if my documentation for changes is not "up to scratch" but does not follow the process himself (as with the most recent case over the long weekend).

 

Over the weekend we had a major change in the back end policy that controls our Citrix environment, used by 95% of the staff in the company. 
There was no communication from the SysOps who put that change in place, and in fact did not even tell his colleagues he had made the change. There was no change put into our change control process, and we are several weeks away from the next scheduled change weekend. 
The change was not tested fully and as a result users called for support, making me very angry when I figured out what had changed.

 

We have all the email systems setup so it takes 15 seconds to notify everyone in the team that a change has been made ...


31 posts

Geek
+1 received by user: 15


  # 1717240 8-Feb-2017 15:33
7 people support this post
Send private message

Pathway 1 = have the CIO/Head of IT/Whoever holds the budget for IT, mandate that for the next unapproved change, a termination of said employee will result. The Nuclear Option...

 

Pathway 2 = find a friendly SysOp, buy him/her coffee and formulate a relationship where they give you a heads-up on changes. The Comrade Option...

 

Pathway 3 = disable the Change Manager's account next time it happens. Don't re-enable it until they agree to tell you about changes. The Hostage Option...

 

 

 

Good Luck!

 

 


2249 posts

Uber Geek
+1 received by user: 698

Subscriber

# 1717429 8-Feb-2017 23:03
Send private message

gzt: There may be another side to the story. From their perspective they probably make lots of changes and very few of those have any user impact.

 

 

 

Some times user scream testing = best testing.


98 posts

Master Geek
+1 received by user: 56


  # 1717460 9-Feb-2017 07:32
One person supports this post
Send private message

Forward the help desk phone to his mobile.



2194 posts

Uber Geek
+1 received by user: 186


  # 1717463 9-Feb-2017 07:41
Send private message

doublek69:

 

Pathway 1 = have the CIO/Head of IT/Whoever holds the budget for IT, mandate that for the next unapproved change, a termination of said employee will result. The Nuclear Option...

 

Pathway 2 = find a friendly SysOp, buy him/her coffee and formulate a relationship where they give you a heads-up on changes. The Comrade Option...

 

Pathway 3 = disable the Change Manager's account next time it happens. Don't re-enable it until they agree to tell you about changes. The Hostage Option...

 

 

 

Good Luck!

 

 

 

 

 

 

How I wish any of these were an option

 

Option 1 - he is "best buds" with my manager, his manager and the big boss and he will never be fired (holds too many other cards close to his chest)

 

Option 2 - the others are just as bad and in the most recent weekend example, they didn't know about the change either

 

Option 3 - more than likely to get me fired

 

 

 

spencer: Forward the help desk phone to his mobile.

 

This probably my only option; check emails after change weekend - if no communication then forward phones (if communication, do job)


907 posts

Ultimate Geek
+1 received by user: 287


  # 1717502 9-Feb-2017 09:32
One person supports this post
Send private message

My unhelpful advice on dealing with problems at work (and this is what I've done in the past):

 

1. Talk with your manager.

 

2. Wait for something to happen.

 

3. If nothing happens, talk with your manager.

 

4. Wait for something to happen.

 

5. Give up waiting (after 6-12 months), and resign.  Citing the problem if your manager isn't a dick who will give you a bad reference.

 

It sounds like you're awfully close to step 5.  It's not one I've taken without having another job lined up.

 

I had to leave a job that was fun and technically challenging, because it caused me far too much stress.  I'd get given urgent problems to fix and I'd move heaven and earth to fix them.  Got sick of the relentless set of urgent issues and my manager said he'd try assigning them to other people.  He did that, the other people would usually give up after a week and I'd have to fix them anyway, now a week more urgent than they were before.


Mr Snotty
8588 posts

Uber Geek
+1 received by user: 4492

Moderator
Trusted
Lifetime subscriber

  # 1717527 9-Feb-2017 10:19
Send private message

How about creating a slack account for the company and asking the person nicely for the benefit of everyone if he puts planned / implemented changes in there? There isn't any need for either office poop chucking or a hostage situation, just explain the business case (less wasted time etc) and be nice about it.

As a DevOPS engineer in my previous roles I always HATED having change management policies but I do see the point of it. At one point one of my peers decided it was a great idea to deviate from the standard system build (Debian based, postgresql) to a new standard of CentOS, MySQL and a whole bunch of security rules. Needless to say this broke many things and the devs got pretty upset (understandable) - when I spoke out and tried to fight it I ended up getting made redundant then asked back 3mo later to help clean up the mess (of which, I declined).




 1 | 2
Filter this topic showing only the reply marked as answer View this topic in a long page with up to 500 replies per page Create new topic



Twitter »

Follow us to receive Twitter updates when new discussions are posted in our forums:



Follow us to receive Twitter updates when news items and blogs are posted in our frontpage:



Follow us to receive Twitter updates when tech item prices are listed in our price comparison site:





News »

HPE to acquire supercomputing leader Cray
Posted 20-May-2019 11:07


Techweek starting around NZ today
Posted 20-May-2019 09:52


Porirua City Council first to adopt new council software solution Datascape
Posted 15-May-2019 12:00


New survey provides insight into schools' technology challenges and plans
Posted 15-May-2019 09:30


Apple Music now available on Alexa devices in Australia and New Zealand
Posted 15-May-2019 09:11


Make a stand against cyberbullying this Pink Shirt Day
Posted 14-May-2019 20:23


Samsung first TV manufacturer to launch the Apple TV App and Airplay 2
Posted 14-May-2019 20:11


Vodafone New Zealand sold
Posted 14-May-2019 07:25


Kordia boosts cloud performance with locally-hosted Microsoft Azure ExpressRoute
Posted 8-May-2019 10:25


Microsoft Azure ExpressRoute in New Zealand opens up faster, more secure internet for Kiwi businesses
Posted 8-May-2019 09:39


Vocus Communications to deliver Microsoft Azure Cloud Solutions through Azure ExpressRoute
Posted 8-May-2019 09:25


Independent NZ feature film #statusPending to premiere during WLG-X
Posted 6-May-2019 22:13


The ultimate dog photoshoot with Nokia 9 PureView #ForgottenDogsofInstagram
Posted 6-May-2019 09:41


Nokia 9 PureView available in New Zealand
Posted 6-May-2019 09:06


Motorola Solutions joins local partners to deliver advanced communications network in New Zealand
Posted 30-Apr-2019 21:50



Geekzone Live »

Try automatic live updates from Geekzone directly in your browser, without refreshing the page, with Geekzone Live now.


Support Geekzone »

Our community of supporters help make Geekzone possible. Click the button below to join them.

Support Geezone on PressPatron



Are you subscribed to our RSS feed? You can download the latest headlines and summaries from our stories directly to your computer or smartphone by using a feed reader.

Alternatively, you can receive a daily email with Geekzone updates.