Predicting ABM Results with Covering Arrays and Random Forests

Abstract

Simulation is a useful and effective way to analyze and study complex, real-world systems. It allows researchers, practitioners, and decision makers to make sense of the inner working of a system that involves many factors often resulting in some sort of emergent behavior. The number of parameter value combinations grows exponentially and it quickly becomes infeasible to test them all or even to explore a suitable subset. How does one then efficiently identify the parameter value combinations that matter for a particular simulation study? In addition, is it possible to train a machine learning model to predict the outcome of an agent-based model (ABM) with a systematically chosen small subset of parameter value combinations? We explore these questions in this paper. We propose utilizing covering arrays to create t-way (t = 2, 3, etc.) combinations of parameter values to significantly reduce an ABM’s parameter value exploration space. In our prior work we showed that covering arrays are useful for systematically decreasing an ABM’s parameter space. We now build on that work by applying it to Wilensky’s Heat Bugs model and training a random forest machine learning model to predict simulation results by using the covering arrays to select our training and test data. Our results show that a 2-way covering array provides sufficient training data to train our random forest to predict three different simulation outcomes. Our process of using covering arrays to decrease parameter space to then predict ABM results using machine learning is successful.