Geekzone: technology news, blogs, forums
Guest
Welcome Guest.
You haven't logged in yet. If you don't have an account you can register now.




1022 posts

Uber Geek
+1 received by user: 56


Topic # 52728 7-Dec-2009 12:30
Send private message

Hey guys,

Have been bashing my head against the wall on this one and need some help.

The setup:

ESXi running guest OS of Windows Server 2003 Standard SP2. This is running terminal services with Office 2003 SP3, IE7.

What happens:

After a few days of use the server begins to run slow, takes a long time to log on etc.
Users complain of hanging apps, slow to open attachement etc

What I have done:

Ran up a copy of Process explorer and found SVCHOST.exe to the be problem, Narrowed it down to Terminal Services taking up the CPU usage.

Read up on some forums about similar issues, tried disabling Windows Update (as suggested but I didn't think it would work due to the Terminal service causing SVCHOST.exe to run high)

Tried to get all users to log off and see if the problem goes away (as in maybe the TS was overloaded) But this made no difference, with only me as the admin logged on the CPU still spiked at 100% for extended periods of time.

So far the only "fix" is to restart the server.  Not ideal to have to do all the time.

Has anyone seen anything like this before?  Or does anyone have any bright ideas for me to try next?

The only thing that possibly is causing the issue would be a rootkit, but I would expect the issue to happen straight after a reboot if that was the case.

Any ideas would be great!

Create new topic
Go Hawks!
782 posts

Ultimate Geek
+1 received by user: 30

Trusted
Subscriber

  Reply # 279972 7-Dec-2009 13:02
Send private message

Do you have enough RAM allocated to the Windows Server 2003 Guest?

We've recently encountered an issue with ESXi running 4 virtual servers - unfortunately the fourth machine was a "quick fix" for testing a problem, but it turned out there just wasn't enough memory installed for running up all four machines - we suffered many a high CPU load problems until we shut down the fourth machine and then reallocated back the memory to the remaining three servers.

SVCHOST.exe also plays host to quite a number of items, and I have seen instances of CPU load being held at 100% by poorly written services (i.e. ones that don't implement sleep() type commands in them if they are to run all the time and only periodically check statuses) (I suspect that's not the case here as you mention that the other users notice slow down on the machine)

Also - what's the hard drive usage like? From memory, windows uses a pagefile as swap - if you're main drive / pagefile drive is filling up it could be that causing a slow down?

HTH.



1022 posts

Uber Geek
+1 received by user: 56


  Reply # 279981 7-Dec-2009 13:21
Send private message

Thanks for the reply wazzageek,

The guest currently has 4GB of RAM installed. When the slowdown happens the FS (also virtualized on the same ESX Host) shows no signs of speed issues.

I have seen issues with guest OS's having issues when 2 vCPU's are assigned to it, but in this case the server was built with only 1 vCPU.

VMWARE have suggested removing serial ports, virtual floppy drives and virtual cd drives - but this doesnt apply in this case as those devices are already removed.

As for the disk access, I did not check that while the server was having issues, but at the moment it is normal. All drives have over 10GB free space too.

Tonight I am going to run a RootKit revealer over the server and see what comes up, also I will run HijackThis and see if that shows anything. The server is locked down so hopefully a user hasn't managed to put a nastie on it!

Cheers

 
 
 
 


1200 posts

Uber Geek
+1 received by user: 3

Trusted

  Reply # 279983 7-Dec-2009 13:25
Send private message


We ran into a similar problem with Hyper-V and disk access being slow as a wet wig, we originally thought it was the Hyper-V emulated IDE Access (install the integration tools be it VMWARE or Hyper-V and make sure they match the hypervisor versions.)

But it turned out to be Antivirus (Trend Micro) which caused the Terminal Server to run like a dog.

Also run RD II Disk benchmark and post the results here

http://download.cnet.com/Rd-II-Disk-Benchmark/3000-2086_4-10657286.html





Tyler - Parnell Geek - iPhone 3G - Lenovo X301 - Kaseya - Great Western Steak House, these are some of my favourite things.



1022 posts

Uber Geek
+1 received by user: 56


  Reply # 279990 7-Dec-2009 13:50
Send private message

exportgoldman:
We ran into a similar problem with Hyper-V and disk access being slow as a wet wig, we originally thought it was the Hyper-V emulated IDE Access (install the integration tools be it VMWARE or Hyper-V and make sure they match the hypervisor versions.)

But it turned out to be Antivirus (Trend Micro) which caused the Terminal Server to run like a dog.

Also run RD II Disk benchmark and post the results here

http://download.cnet.com/Rd-II-Disk-Benchmark/3000-2086_4-10657286.html



Cool, I will run the benchmark tonight along with the other things and post back about it later.

VMware tools is up-to-date, I updated that just over a week ago (after the problem started)

Just on that note, the problem seems to have start without any change on the system.  I know that no windows updates had been applied to break it as I push them out by WSUS every 3 months (part of the contract)

AV you say, I will have a look see, but due to the process that was spiking being terminal services it sounds unlikely, but it always pays to check just in case!



1022 posts

Uber Geek
+1 received by user: 56


  Reply # 280078 7-Dec-2009 18:25
Send private message

Here are the results from the disk benchmark, however during the benchmarking the system was still responsive in terms of CPU requests.

https://cdn.geekzone.co.nz/imagessubs/blog26e664ec822461dc78aae620f0ee8857.jpg  

Will post the results from other scans once they have completed



1022 posts

Uber Geek
+1 received by user: 56


  Reply # 280301 8-Dec-2009 10:03
Send private message

---Update---

Ran some virus and spyware scans last night and removed 17 infections, mostly cookies but some possible nasties as well.

Not sure if it is the ultimate cause of the problem, but it may not have been helping it!

Will keep an eye on it for now and see how things pan out.

194 posts

Master Geek
+1 received by user: 28


  Reply # 280338 8-Dec-2009 11:43
Send private message

One thing that has worked wonders for our 2003 terminal servers is the UPHClean service from MS (From here: http://www.microsoft.com/downloads/details.aspx?FamilyId=1B286E6D-8912-4E18-B570-42470E2F3582&displaylang=en)

Quick overview: 'The User Profile Hive Cleanup service helps to ensure user sessions are completely terminated when a user logs off. System processes and applications occasionally maintain connections to registry keys in the user profile after a user logs off. In those cases the user session is prevented from completely ending.'

As you can imagine this can build up and cause all sorts of issues. I'd highly recommend it - phones pretty much never ringing these days :)



1022 posts

Uber Geek
+1 received by user: 56


  Reply # 280340 8-Dec-2009 11:46
Send private message

CapBBeard: One thing that has worked wonders for our 2003 terminal servers is the UPHClean service from MS (From here: http://www.microsoft.com/downloads/details.aspx?FamilyId=1B286E6D-8912-4E18-B570-42470E2F3582&displaylang=en)

Quick overview: 'The User Profile Hive Cleanup service helps to ensure user sessions are completely terminated when a user logs off. System processes and applications occasionally maintain connections to registry keys in the user profile after a user logs off. In those cases the user session is prevented from completely ending.'

As you can imagine this can build up and cause all sorts of issues. I'd highly recommend it - phones pretty much never ringing these days :)


Yup, it is an awesome tool!  I put it on all TS's as standard since I found it a couple of years ago, helps so much with profile niggles that some users have.

Create new topic



Twitter »

Follow us to receive Twitter updates when new discussions are posted in our forums:



Follow us to receive Twitter updates when news items and blogs are posted in our frontpage:



Follow us to receive Twitter updates when tech item prices are listed in our price comparison site:





News »

New Zealand Ministry of Education chooses Unisys for cloud-based education resourcing management system
Posted 22-Nov-2017 22:00


Business analytics software powers profits for NZ wine producers
Posted 22-Nov-2017 21:52


Pyrios strikes up alliance with Microsoft integrator UC Logiq
Posted 22-Nov-2017 21:51


The New Zealand IT services ecosystem - it's all digital down here
Posted 22-Nov-2017 21:49


Volvo to supply tens of thousands of autonomous drive compatible cars to Uber
Posted 22-Nov-2017 21:46


From small to medium and beyond: Navigating the ERP battlefield
Posted 21-Nov-2017 21:12


Business owners: ERP software selection starts (and finishes) with you
Posted 21-Nov-2017 21:11


Why I'm not an early adopter
Posted 21-Nov-2017 10:39


Netatmo launches smart home products in New Zealand
Posted 20-Nov-2017 20:06


Huawei Mate 10: Punchy, long battery life, artificial intelligence
Posted 20-Nov-2017 16:30


Propel launch Disney Star Wars Laser Battle Drones
Posted 19-Nov-2017 21:26


UFB killer app: Speed
Posted 17-Nov-2017 17:01


The case for RSS — MacSparky
Posted 13-Nov-2017 14:35


WordPress and Indieweb: Take control of your online presence — 6:30 GridAKL Nov 30
Posted 11-Nov-2017 13:43


Chorus reveals technology upgrade for schools, students
Posted 10-Nov-2017 10:28



Geekzone Live »

Try automatic live updates from Geekzone directly in your browser, without refreshing the page, with Geekzone Live now.



Are you subscribed to our RSS feed? You can download the latest headlines and summaries from our stories directly to your computer or smartphone by using a feed reader.

Alternatively, you can receive a daily email with Geekzone updates.