How accurate and powerful are randomization tests in multivariate analysis of variance?

Abstract

Multivariate analysis of variance, based on randomization (permutation) test, has become an important tool for ecological data analyses. However, a comprehensive evaluation of the accuracy and power of available methods is still lacking. This is a thorough examination of randomization tests for multivariate group mean differences. With simulated data, the accuracy and power of randomization tests were evaluated using different test statistics in one-factor multivariate analysis of variance (MANOVA). The evaluations span a wide spectrum of data types, including specified and unspecified (field data) distributional properties, correlation structures, homogeneous to very heterogeneous variances, and balanced an unbalanced group sizes. The choice of test statistic strongly affected the results. Sums of squares between groups (Qb) computed on Euclidean distances (Qb-EUD) gave better accuracy. Qb on Bray-Curtis, Manhattan or Chord distances, the multiresponse permutation procedure (MRPP) and the sum of univariate ANOVA F produced severely inflated type I errors under increasing variance heterogeneity among groups, a common scenario in ecological data. Despite pervasive claims in the ecological literature, the evidence thus suggests caution when using test statistics other than Qb-EUD.