Detecting group differences in sequential association using sampled permutations: Log odds, kappa, and phi compared

Abstract

When determining whether a particular transition is more characteristic of one group than of another two things are required: an index associated with the transition of interest and a statistical test that can determine whether group membership systematically affects values for that index. Here the familiar parametric t test is compared with a test based on sampled permutations. Indices considered are the odds and log odds ratio, Yule's Q, Wampold's (1989) transformed kappa, and phi. The odds and log odds ratio are monotonically increasing functions of Yule's Q and so give similar results. Yule's Q and phi are essentially rank order invariant and usually give similar results. Transformed kappa however, rank orders subjects somewhat differently than the others; moreover, it appears somewhat biased. With respect to the tests, when subjects are 20 or more it does not matter much whether sampled permutation or parametric t tests are used; both yield essentially the same result. However, when subjects are fewer than 20, or whenever there is any other reason to think that parametric assumptions may not be met, permutation tests are recommended. A computer program that effects such tests is described.