Tuesday, August 31, 2004

Presenting Performance Figures

Authors need to be very careful the exact terms they use to describe the performance characteristics of a particular piece of technology. As Simon Robinson , the technical reviewer for my book often pointed out, benchmark results only directly apply to the test they were derived, and are influenced by my factors like hardware, framework version, operating system and other running process.

In my benchmark harness, the test cases are executed many times on a high priority thread to minimize the impact of some of these potentially interferes, but at the end of the day, the test results are only as good as the code inside the benchmarks.

Even for great tests run using the harness, it is critically important that the results are described. Take, for example, virtual methods. In a test for the book, I compared the cost of calling methods with various types of modifiers applied, and found that there was a performance impact for virtual methods. The impact is quite small, and as John Lam points out, depending on the type of modifier and how the processor groups instructions for execution, the performance impact may be eliminated.

Before my book came out, Fawcette published an article by Francesco Balena in which he stated (or did when this link worked):

"Methods that implement interface members are two to seven times slower than regular methods, depending on the language you use and the coding technique you use to declare and call the method."

See the problem? Francesco implies that by being interface-implementing methods, THE METHOD itself will run "two to seven times slower". A software engineer I work with actually stopped using interfaces in his code based on this advice. What Francesco should have said is "Methods that implement interface members are two to seven times slower to call than …". The actual method's execution speed won't be effected, and for a method that does any type of complex processing, the impact will be negligible. Francesco's slight linguistic slip-up meant that this part of the article conveyed a piece of information that was totally wrong. (I'm not trying to pick on Francesco here - he is a great author, and I enjoy has work. I'm just quoting his slip-up because of the interesting results it has caused. I'm sure there are a few places in my writing where I've made the same slip-up).

The motto

Be careful what you write - people take your words literally at times.

Be skeptical of all benchmarks, run the test yourself if in doubt, and always be careful how you apply the result of your benchmark to a specific problem.


Post a Comment

<< Home