The Sky Is Not the Limit

Abstract

We live in the era of Big Data, or at least our awareness of Big Data's presence and impact has sharpened in the past ten years. Compared to data characteristics decades ago, Big Data not only means a deluge of unfiltered bytes, but even more importantly it represents a dramatic increase in data dimensionality (the number of variables) and complexity (the relationships among the often interdependent variables, intricacy of cluster structure). Along with the opportunities for nuanced understanding of processes and for decision making, these data created new demands for information extraction methods in terms of the detail that is expected to be identified in analysis tasks such as clustering, classification, regression, and parameter inference. Many traditionally favored techniques do not meet these challenges if one's aim is to fully exploit the rich information captured by sophisticated sensors and other automated data collection techniques, to ensure discovery of surprising small anomalies, discriminate important, subtle differences, and more. A flurry of technique developments has been spawned, many augmenting existing algorithms with increasingly complex features. © Springer International Publishing Switzerland 2014.