INSTANT: A Runtime Framework to Orchestrate In-Situ Workflows

Abstract

In-situ workflow is a type of workflow where multiple components execute concurrently with data flowing continuously. The adoption of in-situ workflows not only accelerates mission-critical scientific discoveries but also enables responsive disaster predictions. Although there are recent studies on the performance and efficiency aspects of in-situ workflows, the support for portability and distributed computing environments is limited. We present INSTANT, a runtime framework to configure, plan, launch, and monitor in-situ workflows for distributed computing environments. INSTANT provides intuitive interfaces to compose abstract in-situ workflows, manages in-site and cross-site data transfers with ADIOS2, and supports resource planning using profiled performance data. We use two real-world workflows as use cases: a coupled wildfire spreading workflow and a computational fluid dynamics (CFD) workflow coupled with machine learning and visualization. Experiments with the two real-world use cases show that INSTANT effectively streamlines the orchestration of complex in-situ workflows, and its resource planning capability allows INSTANT to plan and carry out efficient in-situ workflow executions under various computing resource availability.