Sector and sphere: The design and of a high-performance data cloud

Abstract

Cloud computing has demonstrated that processing very large datasets over commodity can be done simply, given the right programming model and infrastructure. In paper, we describe the design and implementation of the Sector storage cloud and Sphere compute cloud. By contrast with the existing storage and compute clouds, can manage data not only within a data centre, but also across geographically data centres. Similarly, the Sphere compute cloud supports user-defined (UDFs) over data both within and across data centres. As a special case, style programming can be implemented in Sphere by using a Map UDF by a Reduce UDF. We describe some experimental studies comparing /Sphere and Hadoop using the Terasort benchmark. In these studies, Sector is twice as fast as Hadoop. Sector/Sphere is open source. © 2009 The Royal Society.