Outlier Exclusion Procedures for Reaction Time Analysis: The Cures Are Generally Worse Than the Disease

Abstract

A methodological problem in most reaction time (RT) tasks is that some measured RTs may be outliers, being either too fast or too slow to reflect the task-related processing of interest. Numerous ad hoc procedures have been used to identify these outliers for exclusion from further analyses, but the accuracies of these methods have not been systematically compared. The present study compared the performance of 58 different outlier exclusion procedures (OEPs) using four huge datasets of real RTs. The results suggest that these OEPs are likely to do more harm than good, because they incorrectly identify outliers, increase noise, introduce bias, and generally reduce statistical power. The results suggest that RT researchers should not automatically apply any of these OEPs to clean their RT data prior to the main analyses.