A practical illustration of methods to deal with potential outliers: A multiverse outlier analysis of study 3 from Brummelman, Thomaes, Orobio de Castro, Overbeek, and Bushman (2014)

Abstract

Recently, Brummelman, Thomaes, Orobio de Castro, Overbeek, and Bushman (2014: Study 3) demonstrated that inflated praise benefits challenge seeking of children with high self-esteem, but harms challenge seeking of children with low self-esteem. In the present paper, we examined the original data set on model-fit and prediction outliers according to various reasonable criteria and norms. Subsequently, we carried out a multiverse outlier re-analysis on the data of Brummelman and colleagues’ Study 3, employing the same analytical approach as the original authors did but excluding outliers. Out of the twelve re-analyses in the multiverse, six demonstrated that removing only a small number of outliers rendered the originally reported crucial interaction effect between self-esteem and type of praise non-significant and produced a sizeable reduction of the effect size. The present paper illustrates the use of reporting outlier analyses, which lies in allowing a critical evaluation of the empirical evidence and offering a more complete picture that enhances future studies in the field.