A Bootstrap-Surrogate Approach for Sequential Experimental Design for Simulation Models

Abstract

The bootstrap method is a widely used tool for quantifying the uncertainty associated with a given statistical estimator or machine learning method. This paper proposes a novel approach for sequential experimental design that uses the bootstrap in conjunction with an interpolating surrogate model. Consider the problem of fitting a surrogate to a computationally expensive simulation model that yields a numerical output given the values of a set of continuous input variables. To fit a surrogate, initial data points are obtained by running the simulations at a set of space-filling design points. The proposed Bootstrap-Surrogate method improves on this experimental design by sequentially identifying points where the surrogate prediction uncertainty is high and then evaluating the simulation at those points. The surrogate prediction uncertainty at a candidate simulation point is estimated using a weighted combination of two criteria, one based on the bootstrap standard error of the surrogate predictions at the candidate point and the other based on the minimum distance of the candidate point from previous design points. The method is implemented using a radial basis function (RBF) surrogate and tested on groundwater bioremediation models and several test problems. The results show that the Bootstrap-RBF approach generally yields better experimental design points for surrogate modeling (as measured by RMSE on a large test set) than those obtained by an optimum Latin hypercube sample or a sequential experimental design based on a maximin criterion on the groundwater bioremediation models and on the test problems with similar structures.