End-to-end Integration of Scientific Workflows on Distributed Cyberinfrastructures: Challenges and Lessons Learned with an Earth Science Application

Abstract

Distributed cyberinfrastructures (CI) pose opportunities and challenges for the execution of scientific workflows, especially in the context of Earth science applications. They provide heterogeneous resources that can meet the needs of the applications that are part of the scientific workflows and provide the necessary performance and scalability to achieve scientific goals. However, the challenge with distributed CI is that it is difficult to find the right resources for the applications and to orchestrate the workflow execution from resource provisioning to job execution to delivering the final results. In some cases, poor choice of resources may result in slow execution or outright failure. In this paper, we present Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) Pegasus, a CI solution built as part of the U.S. National Science Foundation ACCESS program that provides automated execution of scientific applications. We demonstrate Pegasus's capabilities with SOil MOisture SPatial Inference Engine (SOMOSPIE), an earth science multi-component application for fine-grained soil moisture predictions. We identify a roadmap to migrate applications such as SOMOSPIE on ACCESS resources with the support of ACCESS Pegasus, outlining both strengths and weaknesses of this approach.