How do researchers choose their goals of inference? A survey experiment on the effects of the state of research and method preferences on the choice between research goals

Abstract

In empirical research, scholars can choose between an exploratory causes-of-effects analysis, a confirmatory effects-of-causes approach, or a mechanism-of-effects analysis that can be either exploratory or confirmatory. Understanding the choice between the approaches is important for two reasons. First, the added value of each approach depends on how much is known about the phenomenon of interest at the time of the analysis. Second, because of the specializations of methods, there are benefits to a division of labor between researchers who have expertise in the application of a given method. In this preregistered study, we test two hypotheses that follow from these arguments. We theorize that exploratory research is chosen when little is known about a phenomenon and a confirmatory approach is taken when more knowledge is available. A complementary hypothesis is that quantitative researchers opt for confirmatory designs and qualitative researchers for exploration because of their academic socialization. We test the hypotheses with a survey experiment of more than 900 political scientists from the United States and Europe. The results indicate that the state of knowledge has a significant and sizeable effect on the choice of the approach. In contrast, the evidence about the effect of methods expertise is more ambivalent.