Estimating the performance of cloud-based systems using benchmarking and simulation in a complementary manner

Abstract

Estimating future runtime performance and cost is an essential task for Chief Information Officers in deciding whether to adopt a Cloud-based system. Benchmarking and simulation are two techniques that have long been practiced towards reliable estimation. Benchmarking involves (potentially) high cost and time consumption, but oftentimes yields more reliable estimates than simulation, while the simulation is much cheaper and faster than benchmarking, but less reliable. In order to deal with this dichotomy, we propose a complementary approach to estimating the performance of Cloud-based systems, whereby performance estimates can be obtained in a fast, inexpensive, and also reliable way. In this approach, the ontological concepts of a benchmark model, whose benchmark results have already been obtained, are mapped into those of a simulation model, while the mismatches and similarities between the two models are taken care of, through measures of similarity between the two. This ontology-driven construction of simulation models is intended not only to yield more reliable simulation results but also to help better explain why the simulation results may, or may not, be reliable. To validate our complementary approach, simulation models are constructed using CloudSim, and the simulation results are compared against the corresponding benchmark results, by using our prototype tool, collected from Amazon Web Service (AWS) and Google Compute Engine (GCE) by using the Yahoo! Cloud Serving Benchmark (YCSB) tool. These experiments show that the simulation results show about 90% accuracy with respect to the benchmark results, and additionally we feel we could better explain why this happens.