Designing hydra with centralized versus decentralized control: A comparative study

Abstract

Today's networked and distributed applications, by and large, rely on cloud services. However, solely cloud-based services are not the ideal solution for all use cases, in particular, the case of high volume data sharing in scientific computing whose cloud usage costs could be prohibitively high. Thus we take on a task of building a distributed, federated data repository, dubbed Hydra, for sharing large volume scientific data. In this paper, we compare two design choices: designing Hydra over TCP/IP with a centralized controller, and designing Hydra over Named Data Network (NDN) to enable distributed control. Our study shows that (i) building Hydra over TCP/IP with a central controller offers a simple, straightforward design; (ii) however, the controller necessarily needs to be replicated for scalability and reliability, and cloud CDN is needed to scale data delivery, both bringing additional complexity into the overall design; and (iii) building Hydra over NDN automatically offers scalable and efficient data dissemination at volume, as well as enables distributed control with high resiliency.