Rancid: Reliable Benchmarking on Android Platforms

Abstract

Benchmarking is an important step in the code optimization process that enables empirical performance evaluations in computer systems. Application profiling allows the detection of bottlenecks within the code, and benchmarking can be used to measure the effect of optimizations on performance and to compare implementations. However, obtaining reliable and reproducible performance metrics on modern mobile platforms is a complex task that is often overlooked. This is necessary to produce scientifically sound experiments. There are several factors that introduce noise on performance measurements. We identify and measure the most relevant set of these factors and design a methodology that enables more reliable performance benchmarking on Android platforms. We also describe our flexible benchmarking framework, Rancid, designed to transparently solve these problems. It enables application developers to quickly obtain reliable performance metrics for their code on a wide set of platforms. The evaluation of our methodology and framework shows an improvement on the behavior of results, successfully providing more precise and reproducible measurements on a range of devices and implementations.