IT volunteer in Bougainville


Repeat after me, “Real geeks interview badly.”

, posted: 23-Dec-2009 00:27

I thought that hiring a new staff member would be easy, our company is doing well, we have lots of big interesting high profile projects in the pipeline and there is a surplus of workers due to these trying times. Boy was I ever wrong. Repeat after me, Real geeks interview badly. We nearly employed someone which 'interviewed' really well had real world experience, skills and certifications flowing over pages of their CV but could not walk three hours in our existing engineers shoes. We nearly missing out on a super smart amazingly fast learner who didn't have 'experience' and was not as confident as the study bunnies which know how to press flesh and smile while spouting off IT war stories to impress the bosses. 

Since we have been bitten before by a bad staff hire, someone who's CV and skills didn't match up in any way to what we saw every day I decided to rig up a technical exam for the position. We needed something our engineers face on a daily basis, as real world as possible, a day in the life so to speak, without the bad Chinese food. I finalised on a exam which consisted of a Exchange Server with database corruption and a client request to enable Outlook Web Access with SSL/Forms Based authentication. We also broke DNS by adding incorrect hosts file entries and blamed it on the old IT guys which we were 'taking' the work over from. The candidates were allowed to use google and the internet for research and troubleshooting just like our engineers do every day and were given a maximum of 3.5 hours to complete this task.

Oh, and it wouldn't be a real world test without the demands of a Account Manager as well, so we also asked them to list out for the Account Manager what's wrong with the server before we 'took over' the responsibility of this server. We expected the candidates to find things like the Antivirus being 5 years out of date, no Windows Updates (ever), the blue screens of death, no UPS, no backups etc. You know, the basics. I wrote a exam document and marking check sheet to make sure we were marking fairly and consistently.

We had about 40 CV's submitted, and whittled that down to about ten, half were telephone interview only and half were in person interviews, that was the plan.

Apart from some fellow turning up late and then talking on his mobile during the interview, they all interviewed well, and one in particular, lets call her Jenny, had pages full of Microsoft qualifications and massive rollout experience overseas, Jenny was the only one qualified and that also had real world experience with Windows Server 2008 and Exchange Server 2007. Jenny was our favourite, our pick. Smart. Driven. Qualified. I imagined her working here, sitting at her desk, fixing the worlds problems, laughing with the other engineers over bad Chinese food.

We were left with 4 top notch candidates. We forgot about the phone candidates, including a guy which we'll call Johnny.

By this stage I had finally got the exam environment and exam ready and invited them all back to sit it. We made one of our existing engineers sit the test to set the bar and setup a remote viewer for the exam server so I could watch them in my office while  they sat the exam, watched how they approached problems, their mythologies, how quick they were moving around, the use of shortcuts and what tools they used. It was amazing, I learnt more about those candidates during that exam time than any face / question time I had with them before or after. Incidentally we did tell them we were watching, but only after the exam finished so they wouldn't panic or feel any more pressure. 

I watched people flounder around and focus on the wrong things, do damage to Active Directory, completely ignore important tools and errors and I also watched others literally learn software on the fly and resolve problems they had never seen, trained for or had qualifications on. We even had a guy which walked out five minutes into the exam after admitting he was out of his depth and bluffed his way in, but again interviewed quite well.

I was stunned. Absolutely shocked. Jenny our star, my pick, our most qualified candidate, floundered around and scored the lowest of all, she didn't try to repair the database, got fixated on benign windows errors a seasoned engineer knows to ignore, keep trying to start the services over and over without actually fixing anything in between and took the longest of everyone before we had to simply stop her as she ran the clock out.

Johnny, the guy we dismissed with not enough skills in the phone interview and had to frantically arrange a interview for along with some others hit it out of the park and bet everyone. He barnstormed through the exam. He took no prisoners while learning exchange server on the fly, figured out how the databases worked, found the eseutil Jenny, our star candidate, ignored completely. He made backups and tried different things with streaming and db files, checked complex raw eseutil logs, had it mastered and he did it in nearly half the time that Jenny took to not fix the database on a program she trained on, was qualified for. That Microsoft said she was up to snuff on.

Johnny bet everyone by a large margin. We nearly missed having the opportunity to have the best, smartest, quickest learning candidate and in a few years geek god, because we believed that those which interview well, that can memorise answers and are good at interviews would perform best in their jobs which involve none of those skills. I nearly made the same mistakes we made in the past hiring staff. That those with the best war stories and interview nous make the best engineers.

What I have actually found out is that the best engineers, are well, the best at engineering. It took us a while to work this out and actually know how to interview for it, but since we have it's opened my eyes to the vast difference in talking the talk and walking the walk, and more importantly in measuring the ability for someone to walk 3 hours in our engineers shoes.

Oh, and as for Johnny, he will be eating bad Chinese food and if your lucky working on your server soon.  



Other related posts:
Spare a Christmas thought to that most loyal member of your team








Comment by freitasm, on 23-Dec-2009 06:46

Just so you know, a "certified" professional is someone who passed an exam. It doesn't actually mean they can think, or do things.

Of course there are lots of "certified" professionals who can do, but don't assume a certification is the promise of a true professional.

I've heard some interesting stories from people in this area...


Comment by crazed, on 23-Dec-2009 07:20

I personally know of one firm in the NZ IT field that has been in that exact situation, employed exam passed, qualification gained, so-called experienced Uni trained persons for positions only to find out they know nothing, but yet have qualifications to say they do, Diploma's, Degree's and Industry Exams.

The company which employes more than 10 staff, now uses tests and realworld situations to sort out those who say they know compared to know who actually know.

The firm actually takes each potential staff member on for 2 days each, and passes them actual client work under supervision ofcourse for them to complete, you would be amazed who actually fails and who passes.

The firm now has 4 staff who hold no qualifications at all in the fields they are employed in, but their knowledge and experience in the field is amazing.

It just goes to show that in some situations, those who do well in interviews and have the best looking CV's don't always make the best staff.


Comment by Peter, on 23-Dec-2009 09:08

Does this apply to the IT industry only, or can medical doctors be employed by virtue of a simple pre-employment competency test? How about nuclear engineers?


Comment by Zippity, on 23-Dec-2009 17:46

The English and grammar of the author (employer??) leaves a lot to be desired.

Sadly, he/she is not as educated as they think they are :(

[EDIT:


Comment by nate, on 23-Dec-2009 22:52

Great post, just shows the benefits of real world testing vs plenty of certifications which aren't worth the paper they are written on.


Comment by Nik, on 24-Dec-2009 00:44

Really? Deleted the whole post aye. Stephen is that you? Stephen Conroy? #nocleanfeed

[MOD EDIT:]
Not stephen, my name is up the TOP RIGHT of the blog, I removed your previous comments as they contained personal attacks, inappropriate comments and just a general rage against someone it appears that isn't even me.

If you want to discuss the subject at hand that is fine.


Comment by Nik, on 24-Dec-2009 02:02

My original response was both relevant and traditionally respectful for a dissenting view. My disappointment lies in your approach not result.


Comment by kontonnz, on 24-Dec-2009 07:11

at work, we developed a similar test that is designed for our apps support teams that last about 3hrs and its basically just a fix a broken web app, so nothing majorly hard. 
What is intersting is we also found candidates who look great on paper and interview well can have extreme difficulties in troubleshooting, some just sit their for the 3hrs reading the help files and some take a look at the exam and walk out, we've hired some good people using this testing. 

Oh yes we initally tried to give people real work that was supervised our HR department was particullarly unhappy with us for that, as it could mean the candidate could demand reparations


Comment by Gigs, on 24-Dec-2009 13:55

Having gone through a fair number of interviews (and apparently I intervewi well lol) it amazes me that a lot of IT positions don't have a partial practical component to them before being weeded out to final candidates.

A lot of qualifications sound great on paper, but i've worked with many poeple who have MSCE's, CCNA's, others, mixtures of etc who when it comes down to setting up something like a username and password in a router or other simple tasks, flounder and become worthless.

Congrats to the deserving employee, but really, feed them on good chinese food, it makes all the difference :)


Comment by Damager, on 24-Dec-2009 17:54

@Zippity

Okay a few gramnatical and spelling errors here and there. I noticed them too. Who gives a sh!t seriously? Stop being ridiculously pedantic.

@exportgoldman

Interesting read. Although not always appropriate in every situation, the ends seems to justify the means in this one.


Comment by Damager, on 24-Dec-2009 18:01

I have just noticed my own spelling error. Even Michael Jordan had his off days!:P


Comment by Nik, on 24-Dec-2009 20:55

@kontonnz I find it interesting you spelled there "their". Are you sure the problem is not just that you don't look good on paper? Majorly... name the Corp kid, I suspect I've never heard of them!


Author's note by exportgoldman, on 26-Dec-2009 09:45


This seems to be a sensitive topic! I wrote this very late at night and yes made some spelling and gramatical errors, but wanted to get the article out as the content I thougth would be of use to people and make a interesting read.

@Zippity I never claim to be educated, in fact I left school after doing badly in sixth form to work in IT in Advertising.

@
freitasm I know about 'paper' MSCE's etc, but my shock was that this candidite had SO SO many (about 10+) certifications and was the worst of all of them.

And to the rest which left comments echoing our results, thankyou for your insights as well, It's good to know we are not alone in our findings. I've had people call it 'creepy' to watch someone from another room, and had some haters e-mailing saying all sorts of stuff, but it's good to know others have had the same experiences.


Comment by ejrg, on 27-Dec-2009 10:29

Exportgoldman you make valid points, however I have found that many company's hiring practices leave a great deal to be desired. The whole process seems so hit and miss. It looks bad when potential employee's interview badly - what about employers that interview badly. I've asked questions such as "what attracted you to the company?" or "what keeps you at this company?" and not had the question answered - even slightly. " Why should I join this company?" gets even more interesting results. Employers are just as bad as employees at interviewing, but because the employer has all the "power", their performance is either ignored or minimized. The quailty of the interviewer determines how "good" the hire will be. Recently I asked a hiring consultant what type of person the company wanted for a job. The response was that it was in the advertisement (and no it wasn't - a description of the skills that were required was listed, but that doesn't describe the type of person they want. The question wasn't understood or answered). Most employers don't state clear what they really want in an applicant - most really don't know. The wife of a friend of mine went for a job - the real requirement was someone who could talk to people, but that was not clearly stated, but she found out and was able to get the job. Another friend was more concerned about people being reliable and willing to stay in a job for an extended period of time.

What you really wanted is someone who could problem solve and wasn't scared by something they have never come across before - someone who could think and was willing to learn. Question is did your interview process look for those attributes or people who interviewed well (as you found out there is a big difference).


Comment by ejrg, on 27-Dec-2009 10:30

Exportgoldman you make valid points, however I have found that many company's hiring practices leave a great deal to be desired. The whole process seems so hit and miss. It looks bad when potential employee's interview badly - what about employers that interview badly. I've asked questions such as "what attracted you to the company?" or "what keeps you at this company?" and not had the question answered - even slightly. " Why should I join this company?" gets even more interesting results. Employers are just as bad as employees at interviewing, but because the employer has all the "power", their performance is either ignored or minimized. The quailty of the interviewer determines how "good" the hire will be. Recently I asked a hiring consultant what type of person the company wanted for a job. The response was that it was in the advertisement (and no it wasn't - a description of the skills that were required was listed, but that doesn't describe the type of person they want. The question wasn't understood or answered). Most employers don't state clear what they really want in an applicant - most really don't know. The wife of a friend of mine went for a job - the real requirement was someone who could talk to people, but that was not clearly stated, but she found out and was able to get the job. Another friend was more concerned about people being reliable and willing to stay in a job for an extended period of time.

What you really wanted is someone who could problem solve and wasn't scared by something they have never come across before - someone who could think and was willing to learn. Question is did your interview process look for those attributes or people who interviewed well (as you found out there is a big difference).


Comment by exportgoldman, on 27-Dec-2009 11:43

@ejrg, You made some really interesting points, and yes I agree employers do interview badly, but I don't think that the employer having the power means their performance is minimized. If the employers don't do well, people leave the company because it's not a good place to work and the company performs badly. Don't get me wrong though, I absolutuly agree with you, I arn't as good at people management and interviewing as I want to be, and I think perhaps a lot of people in my position are as well in IT as we have been shy geeks which now are in management. I originally thought you were wrong about what we wanted, as our original desire was a mid to senior level microsoft geek god, we interviewed on this, and tested on it but thinking about it, you are right as well that we really wanted a quick thinker and learner, which while kinda implied was not front and centre.


Comment by nzsouthernman, on 28-Dec-2009 11:05

Excellent post! I've noted many times that sometimes MCSE's don't have any idea how to be real engineers unless they've done their MCSE as part of a job requirement. (ie forced to by their employer) :)

We've found that when getting to the final interview stage it's best to let the other engineers who will end up working with the interviewee talk to them and spend some time with them. The existing engineers (especially ones who've been working in their fields for quite a few years) can spot the bullsh*ters within a few minutes, and can also spot the quiet, mouselike but knows their stuff ones as well.

A real-world test like yours is a great idea. I think I'll create a set of VM's with a broken Exchange server to fix ready to use for our next hire!


Comment by ejrg, on 29-Dec-2009 14:09

@exportgoldman
I'm not saying you should make your requirements front and centre, make the potential employee earn the place they get, just be aware that your actual requirements may be different to what you initially thought, and interview for those requirements.


Comment by timestyles, on 30-Dec-2009 12:37

This brings up an important question: how do beginning system administrators get practical experience?  If I want to be a programmer, I just start or join up to a project, maybe open source or public domain.  If I want to be an electronics engineer, I just build something complex.  How can someone get real world experience fixing networks etc when they don't have anything to practice on, and haven't got the experience to be employable?


Comment by pjnet, on 30-Dec-2009 15:38

@timestylesI too am struggling to get sys admin practical experience and have been actively looking for work in the field for some time now, all I have done is immersed myself in as much experimentation at home as I can, setting up VM's, servers and networks. I did a Diploma and a number of certifications a year or so ago but have found it doesn't get me anywhere really, everybody just wants experience. If you are looking to go Microsoft the Technet plus subscription is good value, you can download and try out your skills on any of the MS recent systems and software. For a pretty reasonable fee.  That and Virtualbox (or MS VirtualPC) gives you a fair bit to experiment with. Obviously with Linux there is no end of OS's you can download and try out for free...... I think employers want to know you are prepared to and know how to find out how something works, nobody can be an expert on a system they haven't yet seen, even if they have used the same version somewhere else no two setups are the same.


Add a comment

Please note: comments that are inappropriate or promotional in nature will be deleted. E-mail addresses are not displayed, but you must enter a valid e-mail address to confirm your comments.

Are you a registered Geekzone user? Login to have the fields below automatically filled in for you and to enable links in comments. If you have (or qualify to have) a Geekzone Blog then your comment will be automatically confirmed and shown in this blog post.

Your name:

Your e-mail:

Your webpage:

exportgoldman's profile

Tyler Rosolowski
New Zealand