Beyond Group Mean Differences: A Demonstration With Scale Scores in Psychology

Abstract

In this paper, we present a model for comparing groups on scale score outcomes. The model has a number of features that make it desirable for analysis of scale scores. The model is based on ordinal regression, hence it is able to capture the shape of the data even when the data are highly discrete, or display marked ceiling or floor effects. Additionally, the model incorporates hierarchical modelling to create accurate summaries of the differences in the scale scores across groups. Statistically, the model assumes the data are ordinal, and hierarchically estimates the entire distribution of each group using factor smooths. Substantively, the model is capable of: estimating location-based, dispersion-based and ordinal descriptives estimates for each group; estimating the uncertainty about these estimates; and performing pairwise comparisons of the different estimates. The estimation method is Bayesian, however, we have created a GUI-based application that users may install on their computer to run the model, reducing the barrier to applying the method. The application takes in the raw data and user input, runs the model, and returns multiple model-based graphical summaries of patterns in the data, as well as tables for more precise reporting. We also share code that allows users extend the model to additional research contexts.