An architecture for automatic deployment of brown dog services at scale into diverse computing infrastructures

Abstract

Brown Dog is an extensible data cyberinfrastructure, that provides a set of extensible and distributed data conversion and metadata extraction services to enable access and search within unstructured, un-curated and inaccessible research data across different domains of sciences and social science, which ultimately aids in supporting reproducibility of results. We envision that Brown Dog, as a data cyberinfrastructure, is an essential service in a comprehensive cyberinfrastructure which includes data services, high performance computing services and more that would enable scholarly research in a variety of disciplines that today is not yet possi-ble. Brown Dog focuses on four initial use cases, specifically, addressing the conversion and extraction needs in the research areas of ecology, civil and environmental engineering, library and information science, and use by the general public. In this paper, we describe an architecture that supports contribution of data transformation tools from users, and automatic deployment of the tools as Brown Dog services in diverse infrastructures such as cloud or high performance computing (HPC) based on user demands and load on the system. We also present results validating the performance of the initial implementation of Brown Dog.