IT volunteer in Bougainville


The case of the missing Cache…

, posted: 3-Nov-2010 11:12

Or alternatively "how to make your high end RAID array perform worse than a laptop drive". If you have ever wondered how bad the performance on a RAID array performs with no battery backed write cache, the hard numbers are below and it's not pretty. It's not even Pretty woman pretty. Hide your wife and kids people - these numbers get ugly.

I've hit this problem numerous times within the last year, as IBM abandoned Adaptec RAID controllers (rebranded as ServeRAID) and moved to LSI Logic, and IBM were shipping them in the first year without cache batteries. A casual glance over the pricing and product descriptions does not reveal the missing cache backup batteries, and as the new range are more expensive than the old cards, the cache batteries then double the cost of the RAID cards.

HP's kit doesn't allow you to create RAID5 arrays on controllers without a cache battery - but they allow RAID5 arrays to continue running when the cache battery fails. A smarter approach, I reckon.

When you don't have the magic cache battery, the RAID controllers cache lurches into 100% read caching only, like a dodgy gearbox stuck in second. As the numbers below show, this creates horrific performance problems on RAID Arrays, allowing a single desktop hard drive to out perform a 4 disk 10,000 RPM SAS array by up a factor of 21 on write operations. Hope no one brought one of these for Exchange or SQL or Virtualisation!

I'm no hardware guy, sure I've been building clusters and arrays and servers for longer than my belly is round,  but even I was stunned by these numbers - I knew performance would be worse, knew the missing write cache was a performance bottleneck, but didn't expect read operations to also take a hit as well. This doesn't sit right with me, but these are the numbers. I did a santa and checked them thrice, and Big Blue did too. They thought it was a driver issue and made me rebuild the whole server was their total confusion at such low numbers.

Test notes:

1. The desktop hard drive was a 3 year old 7200RPM drive.

2. The Tool used for testing is RD II (I prefer it over iometer)

3. The OS the testing was done on was a clean install of Windows Server 2008 R2 x64 SP0.

 

Single 7200RPM Desktop Hard Drive

image.

 

IBM x3450 R2 With 256MB Cache Backed Battery (4 SAS Drives 10K RPM 2.5")

image

 

IBM x3450 R2 With 256MB no Cache backed battery (4 SAS Drives 10K RPM 2.5")

image

 

HP DL380 G5 with 512MB Cache Backed Battery (3 SAS Drives 10K RPM 2.5")

image

 

Many of these new boxes (as least for us) went in for Virtualization projects, as they were the first of the Nehalem based servers, which love a bit of VM goodness, but we were having quite lumpy performance.

 

We have now rolled out cache batteries to our fleet, and are surprisingly picking up a bit of new business due to other IT companies falling into the same trap and us spotting it.

 

Check your caches out there kids, and be safe on your writes.



Other related posts:
All Xserve'ed out. Apple exits the enterprise server market.
First Impressions of the IBM x3650 M2








Comment by muppet, on 3-Nov-2010 11:40

So the write cache won't work until you have a battery, is that what you're saying?  I have no clue when it comes to RAID.

Those numbers are impressive though - the differences between them I mean.


Comment by muppet, on 3-Nov-2010 12:13

I have a HP Compaq DL360 G3 with two drives in a RAID1 configuration.

How do I test if it has batteries?  Would it make a different if I did?

Thanks!


Comment by Tyler Rosolowski, on 3-Nov-2010 13:01

Muppet - yes on most if not all controllers write cache is disabled without a battery backed up cache (HP Call them BBWC = Battery Backed Write Cache.)

As you can see from the figures above it makes a hell of a difference, download RD II and see what figures you are getting compared to similar config's above.

http://download.cnet.com/Rd-II-Disk-Benchmark/3000-2086_4-10657286.html?tag=mncol;1

RD II Needs .NET 2.0 installed and I leave the settings at default and just double click on the drive to test (e.g. C:\ drive) to test the speed. You will quickly see in the write column if you have a problem.


Comment by muppet, on 3-Nov-2010 13:24

Server is already running Linux :)

I will have a play with hdparm or sdparm and see if I can get some figures.

How do I put a battery in it?  Just open it up and pop one in?


Comment by gjm, on 3-Nov-2010 13:48

I had a one of my batteries die on me and, yup, performance was absolutely destroyed. Got Mr IBM in to replace my battery and suggested replacing another one that we had that was the same age. He said he couldn't as it wasnt faulty. Guess what happened only 2 days later.....yup.


Comment by gjm, on 3-Nov-2010 13:49


I had a one of my batteries die on me and, yup, performance was absolutely destroyed. Got Mr IBM in to replace my battery and suggested replacing another one that we had that was the same age. He said he couldn't as it wasnt faulty. Guess what happened only 2 days later.....yup.


Comment by rphenix, on 16-Nov-2010 09:33

Good to see 3ware card's arent the only ones with absolute crap performance without a BBU.


Add a comment

Please note: comments that are inappropriate or promotional in nature will be deleted. E-mail addresses are not displayed, but you must enter a valid e-mail address to confirm your comments.

Are you a registered Geekzone user? Login to have the fields below automatically filled in for you and to enable links in comments. If you have (or qualify to have) a Geekzone Blog then your comment will be automatically confirmed and shown in this blog post.

Your name:

Your e-mail:

Your webpage:

exportgoldman's profile

Tyler Rosolowski
New Zealand