Terminal Server 2003 High CPU usage

Forums › Microsoft Windows › Terminal Server 2003 High CPU usage

jaymz

1136 posts

Uber Geek
+1 received by user: 76

#52728 7-Dec-2009 12:30

Hey guys,

Have been bashing my head against the wall on this one and need some help.

The setup:

ESXi running guest OS of Windows Server 2003 Standard SP2. This is running terminal services with Office 2003 SP3, IE7.

What happens:

After a few days of use the server begins to run slow, takes a long time to log on etc.
Users complain of hanging apps, slow to open attachement etc

What I have done:

Ran up a copy of Process explorer and found SVCHOST.exe to the be problem, Narrowed it down to Terminal Services taking up the CPU usage.

Read up on some forums about similar issues, tried disabling Windows Update (as suggested but I didn't think it would work due to the Terminal service causing SVCHOST.exe to run high)

Tried to get all users to log off and see if the problem goes away (as in maybe the TS was overloaded) But this made no difference, with only me as the admin logged on the CPU still spiked at 100% for extended periods of time.

So far the only "fix" is to restart the server. Not ideal to have to do all the time.

Has anyone seen anything like this before? Or does anyone have any bright ideas for me to try next?

The only thing that possibly is causing the issue would be a rootkit, but I would expect the issue to happen straight after a reboot if that was the case.

Any ideas would be great!

wazzageek

1095 posts

Uber Geek
+1 received by user: 108

ID Verified

Trusted

Lifetime subscriber

#279972 7-Dec-2009 13:02

Do you have enough RAM allocated to the Windows Server 2003 Guest?

We've recently encountered an issue with ESXi running 4 virtual servers - unfortunately the fourth machine was a "quick fix" for testing a problem, but it turned out there just wasn't enough memory installed for running up all four machines - we suffered many a high CPU load problems until we shut down the fourth machine and then reallocated back the memory to the remaining three servers.

SVCHOST.exe also plays host to quite a number of items, and I have seen instances of CPU load being held at 100% by poorly written services (i.e. ones that don't implement sleep() type commands in them if they are to run all the time and only periodically check statuses) (I suspect that's not the case here as you mention that the other users notice slow down on the machine)

Also - what's the hard drive usage like? From memory, windows uses a pagefile as swap - if you're main drive / pagefile drive is filling up it could be that causing a slow down?

HTH.

jaymz

1136 posts

Uber Geek
+1 received by user: 76

#279981 7-Dec-2009 13:21

Thanks for the reply wazzageek,

The guest currently has 4GB of RAM installed. When the slowdown happens the FS (also virtualized on the same ESX Host) shows no signs of speed issues.

I have seen issues with guest OS's having issues when 2 vCPU's are assigned to it, but in this case the server was built with only 1 vCPU.

VMWARE have suggested removing serial ports, virtual floppy drives and virtual cd drives - but this doesnt apply in this case as those devices are already removed.

As for the disk access, I did not check that while the server was having issues, but at the moment it is normal. All drives have over 10GB free space too.

Tonight I am going to run a RootKit revealer over the server and see what comes up, also I will run HijackThis and see if that shows anything. The server is locked down so hopefully a user hasn't managed to put a nastie on it!

Cheers

exportgoldman

1202 posts

Uber Geek
+1 received by user: 3

Trusted

#279983 7-Dec-2009 13:25

We ran into a similar problem with Hyper-V and disk access being slow as a wet wig, we originally thought it was the Hyper-V emulated IDE Access (install the integration tools be it VMWARE or Hyper-V and make sure they match the hypervisor versions.)

But it turned out to be Antivirus (Trend Micro) which caused the Terminal Server to run like a dog.

Also run RD II Disk benchmark and post the results here

http://download.cnet.com/Rd-II-Disk-Benchmark/3000-2086_4-10657286.html

Tyler - Parnell Geek - iPhone 3G - Lenovo X301 - Kaseya - Great Western Steak House, these are some of my favourite things.

jaymz

1136 posts

Uber Geek
+1 received by user: 76

#279990 7-Dec-2009 13:50

exportgoldman:
We ran into a similar problem with Hyper-V and disk access being slow as a wet wig, we originally thought it was the Hyper-V emulated IDE Access (install the integration tools be it VMWARE or Hyper-V and make sure they match the hypervisor versions.)

But it turned out to be Antivirus (Trend Micro) which caused the Terminal Server to run like a dog.

Also run RD II Disk benchmark and post the results here

http://download.cnet.com/Rd-II-Disk-Benchmark/3000-2086_4-10657286.html

Cool, I will run the benchmark tonight along with the other things and post back about it later.

VMware tools is up-to-date, I updated that just over a week ago (after the problem started)

Just on that note, the problem seems to have start without any change on the system. I know that no windows updates had been applied to break it as I push them out by WSUS every 3 months (part of the contract)

AV you say, I will have a look see, but due to the process that was spiking being terminal services it sounds unlikely, but it always pays to check just in case!

jaymz

1136 posts

Uber Geek
+1 received by user: 76

#280078 7-Dec-2009 18:25

Here are the results from the disk benchmark, however during the benchmarking the system was still responsive in terms of CPU requests.

https://cdn.geekzone.co.nz/imagessubs/blog26e664ec822461dc78aae620f0ee8857.jpg

Will post the results from other scans once they have completed

jaymz

1136 posts

Uber Geek
+1 received by user: 76

#280301 8-Dec-2009 10:03

---Update---

Ran some virus and spyware scans last night and removed 17 infections, mostly cookies but some possible nasties as well.

Not sure if it is the ultimate cause of the problem, but it may not have been helping it!

Will keep an eye on it for now and see how things pan out.

Dell

Shop now for Dell laptops and other devices (affiliate link).

CapBBeard

211 posts

Master Geek
+1 received by user: 14

#280338 8-Dec-2009 11:43

One thing that has worked wonders for our 2003 terminal servers is the UPHClean service from MS (From here: http://www.microsoft.com/downloads/details.aspx?FamilyId=1B286E6D-8912-4E18-B570-42470E2F3582&displaylang=en)

Quick overview: 'The User Profile Hive Cleanup service helps to ensure user sessions are completely terminated when a user logs off. System processes and applications occasionally maintain connections to registry keys in the user profile after a user logs off. In those cases the user session is prevented from completely ending.'

As you can imagine this can build up and cause all sorts of issues. I'd highly recommend it - phones pretty much never ringing these days :)

jaymz

1136 posts

Uber Geek
+1 received by user: 76

#280340 8-Dec-2009 11:46

CapBBeard: One thing that has worked wonders for our 2003 terminal servers is the UPHClean service from MS (From here: http://www.microsoft.com/downloads/details.aspx?FamilyId=1B286E6D-8912-4E18-B570-42470E2F3582&displaylang=en)

Quick overview: 'The User Profile Hive Cleanup service helps to ensure user sessions are completely terminated when a user logs off. System processes and applications occasionally maintain connections to registry keys in the user profile after a user logs off. In those cases the user session is prevented from completely ending.'

As you can imagine this can build up and cause all sorts of issues. I'd highly recommend it - phones pretty much never ringing these days :)

Yup, it is an awesome tool! I put it on all TS's as standard since I found it a couple of years ago, helps so much with profile niggles that some users have.