Interpretation of significance levels by psychological researchers: The .05 cliff effect may be overstated

Abstract

Comments about previous studies indicate that the interpretation of significance levels by psychological researchers is unequivocally dictated by a binary decision-making framework. In particular confidence in ap level would drop abruptly just beyond the fateful .05 level (cliff effect). A replication of Rosenthal and Gaito's (1963) experiment on the degree of confidence in p levels shows that these claims should be moderated. Detailed analysis of individual curves reveals that the attitude of researchers toward p values is far from being as homogeneous as might be expected. However, most psychological researchers in our study rated graduated confidence judgments, as either exponential or linear. Only a minority of all-or-none respondents exhibited an abrupt drop in confidence.