Since the debut of the Pocket PC in April 2000, media and Pocket PC enthusiasts have had no effective way to measure Pocket PC performance. Several simplistic benchmarking tools have emerged, but most have focused on a limited range of synthetic CPU and graphics tests that have little bearing on real-world performance. Battery testing was performed through error-prone manual methods or using battery testing applications that often had unpredictable results. With the launch of Windows Mobile 2003 for Pocket PC, the product has entered a third generation with new high-water marks for performance and reliability. Now, with Spb Benchmark, reviews will have the proper tool to truly get a handle on device performance and speak with authority, armed with hard metrics.
VP of Marketing and long-time Pocket PC user Jason Dunn had this to say about the release of Spb Benchmark: "As someone who has been using and reviewing every generation of Pocket PC since they first launched, I'm excited to finally have a tool, as a reviewer, that I can run on any ARM-based Pocket PC and get benchmarks that I can trust. The breadth of tests ensures that all important elements of the Pocket PC are tested, and by using XML we're able to quickly export the data and give the reviewer tools to visually compare the results. This tool will truly revolutionize the way performance is discussed in Pocket PC reviews."
Spb Benchmark tests are divided into six separate categories of tests. The first, file system tests, measures how quickly a device can read a 1 MB file, write a 1 MB file, copy 1 MB file, list a directory with 2000 files, read internal databases, write a 10 KB file 100 times, and several other file-based tests. The graphics tests are designed to measure target Pocket PC graphic subsystem performance: DDB BitBlt speed is measured, along with DIB BitBlt and GAPI BitBlt. The built-in application tests provide real-world speed tests based on tasks the average user will perform every day: Pocket Word document loading, Pocket Internet Explorer HTML page load and JPEG load, and the File Explorer listing a folder with many files in it. Other real-world tests include measuring how long it takes the Pocket PC to compress a 1 MB file using ZIP, decompress a 1024 x 768 JPEG photo, the frames-per-second speed in the Spb game Arkaball, and copy a 1 MB file using memcpy(). Other synthetic tests include three CPU tests measuring MFLOPS, MOPS, and MWIPS. ActiveSync speeds are also tested, giving the reviewer a solid grasp of how fast the Pocket PC can move data to and from the desktop PC.
In addition to application-based performance, Pocket PCs tested with Spb Benchmark will undergo storage card tests and battery tests. Have you ever wondered if those "high speed" memory cards really make a difference when using in a Pocket PC? Now you can know for sure with tests like writing 1 MB file, reading a 1 MB file, copying a 1 MB file to and from the storage card, copying a 10 KB file 100 times, and more. Battery tests include having no backlight with zero utilization, video playback with maximum backlight, playing back MP3 files with the screen turned off, maximum backlight with a standard application use test, and maximum backlight with GPRS, Bluetooth or WiFi turned on.
Once a reviewer has completed the tests, they simply connect the Pocket PC to the desktop computer, click a button on the desktop software component and the results are displayed as a comparison chart. Reviewers can compare their test device to our constantly growing database of different Pocket PCs, and bar graphs are automatically generated based on which devices and tests the reviewer wants to highlight.
Spb Benchmark is free for a limited range of users: if you are a member of the media or someone who reviews Pocket PCs for an enthusiast site, we welcome your use of this tool. If you are an end user who wants to learn more about Pocket PC performance using this tool, it is likewise free. If you are a business user who will derive benefit from Spb Benchmark (OEM, ODM, enterprise user evaluating Pocket PC deployment, etc.), Spb Benchmark must be purchased for $495 US.