Against the “one method fits all data sets” philosophy for comparison studies in methodological research

Abstract

Many methodological comparison studies aim at identifying a single or a few “best performing” methods over a certain range of data sets. In this paper we take a different viewpoint by asking whether the research question of identifying the best performing method is what we should be striving for in the first place. We will argue that this research question implies assumptions which we do not consider warranted in methodological research, that a different research question would be more informative, and how this research question can be fruitfully investigated.