Comparing Differences in Accuracy across Conditions or Individuals: An Argument for the Use of Log Odds

Abstract

In experimental psychology, the degree of difference between the proportions of correctly solved items on two related tests (such as word lists) has been calculated by different methods, for example, a simple difference (e.g. as used in within-subjccts ANOVAs), difference relative to potential gain, quotient, difference between standardized z-scores, or by Signal Detection Theory's d′ every one of which may yield different results. The present article discusses the choice of methods with an example from reading research concerned with contextual facilitation in readers with different abilities. Assuming that the total number of correctly solved items captures all relevant variance in subjects' abilities (i.e. it is a sufficient measure), it is demonstrated that the logarithm of the quotient between odds for the frequencies of correct responses (log odds) is the most suitable method of calculation. For example, calculations based on log odds provide an appropriate ranking of the subjects, from the subject with the smallest difference between the two tests to the subject with the largest difference, in accordance with item response theory models.