Analysis of performance variability in public cloud computing

Abstract

Recent reports on cloud computing performance have provided anecdotal evidence to suggest that it varies over time and according to the vendor. This variability could potentially detract from customer experience, and/or result in increased costs to cloud consumers. In order to investigate performance variability over time, this research measured a number of different services provided by five of the major vendors over one month period. The same benchmarks were executed against a dedicated non-cloud system to establish a baseline. Preliminary results showed that not all vendors offer consistent performance over time. While the performance in disk access was relatively stable, high variability in RAM latency and CPU performance was observed in some cases. They suggest that performance variability should be considered when evaluating vendors, perhaps even incorporated to SLAs. This would ensure that vendors are providing the performance stability to match the resources provided, enabling adequate capacity planning and cost control.