Cloudweaver: Adaptive and Data-Driven Workload Manager for Generic Clouds

Abstract

Cloud computing denotes the latest trend in application development for parallel computing on massive data volumes. It relies on clouds of servers to han- dle tasks that used to be managed by an individual server. With cloud computing, software vendors can provide business intelligence and data analytic services for internet scale data sets. Many open source projects, such as Hadoop, offer various software components that are essential for building a cloud infrastructure. Current Hadoop (and many others) requires users to configure cloud infrastructures via pro- grams and APIs and such configuration is fixed during the runtime. In this chapter, we propose aworkload manager (WLM), called CloudWeaver,which provides auto- mated configuration of a cloud infrastructure for runtime execution. The workload management is data-driven and can adapt to dynamic nature of operator through- put during different execution phases. CloudWeaver works for a single job and a workload consisting ofmultiple jobs running concurrently, which aims at maximum throughput using a minimum set of processors.