Applying statistics to the gatekeeping of expert evidence: Introducing the Structured Statistical Judgement (SSJ)

Abstract

Daubert required judges to base their decisions about the admissibility of expert witness testimony in large part on the reliability and validity of empirical observations. Because judges have a wide array of duties and may not be equipped to understand the complexities of statistical analysis, some jurists have recommended that court-appointed experts assist judges in their gatekeeping function. To assist such experts in scrutinizing empirical papers, we propose a Structured Statistical Judgement (SSJ) that takes advantage of advances in the various statistical methods – such as effect sizes that adjust for error – which have allowed researchers to report increasingly more reliable and valid observations. We also include supplementary materials that court-appointed experts can use both as a codebook to operationalize the SSJ and as a quick reference that will aid consultation with judges. An initial application of the SSJ examined all 93 empirical articles published in Psychology, Public Policy, and Law and Law and Human Behavior in 2015 and resulted in excellent interrater reliability (π = 0.83; π = 0.95; π = 0.97), at the same time it indicated that a majority of the articles fail to include the comprehensive and transparent statistical analysis that would be most useful to courts.