Cloud resource management: towards efficient execution of large-scale scientific applications and workflows on complex infrastructures

Abstract

Cloud computing evolved from the concept of utility computing, which is defined as the provision of computational and storage resources as a metered service. Another key characteristic of cloud computing is multitenancy, which enables resource and cost sharing among a large pool of users. Characteristics such as multitenancy and elasticity perfectly fit the requirements of modern data-intensive research and scientific endeavors. In parallel, as science relies on the analysis of very large data sets, data management and processing must be performed in a scalable and automated way. Workflows have emerged as a way to formalize and structure data analysis, thus becoming an increasingly popular paradigm for scientists to handle complex scientific processes. One of the key enablers of this conjunction of cloud computing and scientific workflows is resource management. However, several issues related to data-intensive loads, complex infrastructures such as hybrid and multicloud environments to support large-scale execution of workflows, performance fluctuations, and reliability, pose as challenges to truly position clouds as viable high-performance infrastructures for scientific computing. This paper presents a survey on cloud resource management that provides an extensive study of the field. A taxonomy is proposed to analyze the selected works and the analysis ultimately leads to the definition of gaps and future challenges to be addressed by research and development.