Why Speclnt95 should not be used to benchmark embedded systems tools

Abstract

The SpecInt95 benchmark suite is often used to evaluate the performance of programming tools, including those used for embedded systems programming. Embedded applications, however, are often targeting 8- or 16-bit processors with limited functionality, whereas SpecInt95 has no particular target architecture and a bias towards 32-bit systems. Hence, there are reasons to question the use of SpecInt95 for the evaluation of tools for embedded systems. We present a comparative study of the static properties of a set of embedded application and the SpecInt95 benchmarks. The properties studied include: variable types, function argument lists, type of operations, and the use of local and global memory. The study provides clear evidence that embedded applications and the SpecInt95 program suite differs significantly in several important areas. Hence, we conclude that using SpecInt95 to evaluate or compare tools for embedded systems is likely to be irrelevant or misleading, and that there is a clear need for a benchmark suite tailored for the embedded applications area.