Some Right Ways to Analyze (Psycho)Linguistic Data

Abstract

Much has been written on the abuse and misuse of statistical methods, including p values, statistical significance, and so forth. I present some of the best practices in statistics using a running example data analysis. Focusing primarily on frequentist and Bayesian linear mixed models, I illustrate some defensible ways in which statistical inference¤mdash¤specifically, hypothesis testing using Bayes factors versus estimation or uncertainty quantification¤mdash¤can be carried out. The key is to not overstate the evidence and to not expect too much from statistics. Along the way, I demonstrate some powerful ideas, including the use of simulation to understand the design properties of one's experiment before running it, visualization of data before carrying out a formal analysis, and simulation of data from the fitted model to understand the model's behavior.