Better Big Data via Data Farming Experiments

Abstract

The term ‘big data’ has become intertwined with ‘data mining’ in the minds of many people. Modern computing can generate massive amounts of data via simulation studies, but a key drawback to the data mining paradigm is that it relies on observational data and thus limits the types of insights that can be gained. We can do much better with ‘data farming,’ a metaphor that captures the notion of purposeful data generation from simulation models. Prospective designs of experiments can establish causal relationships, in contrast to data mining that can only find correlations. The use of large-scale designed experiments lets us grow simulation output efficiently and effectively, and is a game changer in terms of the power and flexibility it offers analysts and decision makers. When combined with modern simulation tools and cluster computing , it allows studies to focus on much broader questions and obtain much richer insights. In this chapter, we discuss the implications data farming has on model building, verification and validation , input modeling and data requirements, multi-criteria decision-making and tradeoff analysis , and a few ethical aspects of the decision process.