Trusting floating point benchmarks - are your benchmarks really data independent?

Abstract

Benchmarks are important tools for studying increasingly complex hardware architectures and software systems. Two seemingly common assumptions are that the execution time of floating point operations do not change much with different input values, and that the execution time of a benchmark does not vary much if the input and computed values do not influence any branches. These assumption do not always hold. There is significant overhead in handling denormalized floating point values (a representation automatically used by the CPU to represent values close to zero) on-chip on modern Intel hardware, even if the program can continue uninterrupted. We have observed that even a small fraction of denormal numbers in a textbook benchmark significantly increases the execution time of the benchmark, leading to the wrong conclusions about the relative efficiency of different hardware architectures and about scalability problems of a cluster benchmark. © Springer-Verlag Berlin Heidelberg 2007.