Comparing discrete distributions when the sample space is small

Abstract

This paper describes two new methods for comparing two independent, discrete distributions, when the sample space is small, using an extension of the Storer-Kim method for comparing independent binomials. These methods are relevant, for example, when comparing groups based on a Likert scale, which was the motivation for the paper. In essence, the goal is to test the hypothesis that the cell probabilities associated with two independent multinomial distributions are equal. Both a global test and a multiple comparison procedure are proposed. The small-sample properties of both methods are compared to four other techniques via simulations: Cliff's generalization of the Wilcoxon-Mann-Whitney test that effectively deals with heteroscedasticity and tied values, Yuen's test based on trimmed means, Welch's test and Student's t test. For the simulations, data were generated from beta-binomial distributions. Both symmetric and skewed distributions were used. The sample space consisted of the integers 0(1)4 or 0(1)10. For the global test that is proposed, when testing at the 0.05 level, simulation estimates of the actual Type I error probability ranged between 0.043 and 0.059. For the new multiple comparison procedure, the estimated family wise error rate ranged between 0.031 and 0.054 for the sample space 0(1)4. But for 0(1)10, the estimates dropped as low as 0.016 in some situations. Given the goal of comparing means, Student's t is well known to have practical problems when distributions differ. Similar problems are found here among the situations considered. No single method dominates in terms of power, as would be expected, because different methods are sensitive to different features of the distributions being compared. But in general, one of the new methods tends to have relatively good power based on both simulations and experience with data from actual studies. If, however, there is explicit interest in comparing means, rather than comparing the cell probabilities, Welch's test was found to perform well. The new methods are illustrated using data from the Well-Elderly Study where the goal is to compare groups in terms of depression and the strategies used for dealing with stress.