Familywise type I error of ANOVA and ANOVA on ranks in factorial experiments

Abstract

This research evaluated the importance of a preliminary general analysis of variance (ANOVA) in the interpretation of data from factorial experiments under total nullity. For this, we evaluated the familywise type I error rate (accumulated FWER) of the F test for the unfolding of factorial ANOVA and factorial ANOVA on ranks, which were compared with the FWER for the global effect of treatments. In addition, we evaluated the FWER of the Tukey’s test under total nullity for factorial experiments in the presence or absence of preliminary ANOVA protection (omnibus F test). The study was conducted by simulating data from 2,000 experiments, which were separated into four representative agricultural research scenarios. For both the parametric factorial ANOVA and the non-parametric factorial ANOVA, the FWER significantly exceeded the nominal level of 5%, even under total nullity. While the tests that control the total FWER in the factorials are not being used, the factorial ANOVA should not be performed without the preliminary ANOVA F test showing a significant effect. This, of course, does not apply to tests that are not multiple comparisons, such as Bonferroni, Dunn-Sidak and others, which do not need ANOVA protection. The same recommendation applies to the factorial ANOVA on ranks.