Large scale data analysis

Abstract

As data sets grow in size and complexity, global analysis methods do not necessarily characterize the phenomena of interest, and scientists are increasingly reliant on feature-based analysis methods to study the results of large-scale simulations. This chapter presents a framework that efficiently encodes the set of all possible features in a hierarchy that is augmented with attributes, such as statistical moments of various scalar fields. The resultingmeta-data generated by the framework is orders of magnitude smaller than the original simulation data, yet it is sufficient to support a fully flexible and interactive analysis of the features, allowing for arbitrary thresholds, providing per-feature statistics, and creating various global diagnostics such as Cumulative Density Functions (CDFs), histograms, or time-series. The analysis is combined with a rendering of the features in a linked-view browser that enables scientists to interactively explore, visualize, and analyze data resulting from petascale simulations. While there exist a number of potential feature hierarchies that can be used to segment the simulation domain, we provide a detailed description of two: the merge tree and the Morse-Smale (MS) complex, and demonstrate the utility of this new framework in practical settings.