Creating portable, repeatable, realistic benchmarks for embedded systems and the challenges thereof

Abstract

To appreciate the challenges of analysing embedded processor behaviour, step back in time to understand the evolution of embedded processors. Only a few decades ago, embedded processors were relatively simple devices (compared to today), represented by a host of 8- And 16-bit microcontrollers, and 32-bit microprocessors, with minimal integration. Today, these processors (even the so-called, low-end microcontrollers), have evolved into highly- integrated SoCs with a wide variety of architectures capable of tackling both specific and general-purpose tasks. Associated with these transformations, the benchmarks used to quantify the capabilities have also grown in complexity and range. At the simplest level, benchmarks such as CoreMark analyse the fundamental processor cores. At the other end of the spectrum, system benchmarks, such BrowsingBench, analyse the entire SoC as well as the system software stack and even the physical interfaces. This paper examines some of the challenges of applying such benchmarks, and explains the methodologies used at EEMBC to manage portability, repeatability, and realism. Copyright © 2012 ACM.