Racing with a fixed budget and a self-adaptive significance level

Abstract

F-Race is an offline parameter tuning method which efficiently allocates samples in order to identify the parameter setting with the best expected performance, out of a given set of parameter settings. Using non parametric statistical tests, F-Race discards parameter settings which perform significantly worse than the current best, allowing the surviving parameter settings to be tested on more instances and hence obtaining better estimates for their performance. The statistical tests require setting significance levels which directly affect the algorithm's ability of detecting the best parameter setting, and the total runtime. In this paper, we show that it is not straightforward to set the significance level and propose a simple modification to automatically adapt the significance level such that the failure rate is minimized. This is tested empirically using data drawn from probability distributions with pre-defined characteristics. Results indicate that, under a strict computational budget, F-Race with online adaptation performs significantly better than its counterpart with even the best fixed value. © 2013 Springer-Verlag.