Computer benchmarks and other high-tech baloney

by  Alan Zisman (c) 2006 First published in Business in Vancouver February 7-13, 2006; issue 850

High Tech Office column; 

Benjamin Disraeli is famous for claiming there were three kinds of lies: lies, damn lies and statistics. In the High Tech Office we might want to replace "statistics" with "benchmarks." The idea behind computer benchmarks is noble: to have an objective test making it possible to compare different models of computer hardware independent from the software each is running. The problem is that no one runs a computer without software, so benchmark data offers at best a fun-house reflection of the real world.

On January 10, Apple CEO Steve Jobs announced his company's first Macintosh models built using Core Duo processors from Intel. He boasted of benchmark tests showing the new models to be anywhere from twice to four times as fast as their PowerPC-powered predecessors. While Jobs did the right thing in pointing out that real-world performance might not be quite so fast, most of the media ignored the fine print and highlighted the promise of super speed. A few things to note:

As the name Core Duo suggests, the new Intel CPU has two processors in a single unit. This promises twice the performance of a single CPU, which is reflected by the benchmark tests. But users will see improvement only on applications that have been rewritten for "multi-threading" - the ability to divide tasks between the two processors. Ordinary applications won't see much, if any, improvement.

A computer processor's speed and power get all the hype, but those measurements are only one factor in the performance a user sees.

Hard drive and memory access speed and video capabilities all affect real-world performance. The new iMacs have improvements in these areas, but nowhere near the 2x - 4x speed improvements being touted. For best performance, software code needs to be compiled into programs that are optimized for the hardware. Apple has reworked its operating system and many of its own programs to make good use of the new Intel processors, but most third-party software, including commonly-used programs like Microsoft Office and Adobe Photoshop, have yet to be updated for the new hardware. Intel Macs can run them using built-in translation software called Rosetta, but the translation comes with a speed penalty; these programs feel sluggish, seeming to run at about half the speed they would on a recent generation PowerPC iMac. The result: the new hardware runs software optimized for it faster than the models they replace, but runs other programs slower than last-year's models.

In a few months, more software will be supporting the new Intel-powered Macs. Users should then see the promised blazing performance. In the meantime, however, anyone buying a new Mac to see the super speed promised by the benchmark tests is going to be sadly disappointed.