Inadvertent use of ANOVA in educational research: ANOVA is not a surrogate for MANOVA

Abstract

ANOVA and MANOVA address different research questions and decision on conducting one or the other of these tests relies on the research purpose. One prominent illegitimate analysis of multivariate data is developed out of conducting multiple ANOVAs rather than conducting a MANOVA. Another common mistake about MANOVA applications is the use of improper post hoc procedure. Post hoc procedures are needed to determine why the null hypothesis was rejected. Although the correct post hoc procedure for MANOVA is descriptive discriminant analysis (DDA), many researchers fail to conduct DDA to interpret their MANOVA results. The purpose of this study is two-fold; (1) we aim to emphasize the theory behind the MANOVA and its appropriate post hoc procedure and make clear distinction between surrogate statistical procedures such as ANOVA; and (2) this study also investigates the extent of incorrect analysis of multivariate dependent variables in educational research in Turkey. First, we provided a small simulation study to demonstrate the extent to which multiple ANOVAs yields contradictory results when they are inadvertently used to test group mean differences on multiple dependent variables. Results of the simulations indicated that MANOVA and multiple ANOVAs had severe disagreements under many conditions. Disagreement rate is elevated under the conditions where MANOVA retains the null hypothesis. Then, we systematically reviewed the archives of three education journals, which are classified as higher-, medium, and lower quality journals. Results indicated that correct use of MANOVA with its proper post hoc procedure is not common practice across educational researchers who publish in Turkish education journals.