t-Test: The Good, the Bad, the Ugly, & the Remedy

Abstract

The t-test is one of the most commonly used significance tests to assess whether the means of two groups are statistically significantly different from each other. The use of t-test has become a natural choice, and rarely practitioners question its appropriateness. This article reviews and discusses t-test’s value in providing a rough comparison of two means (the good), its inability to provide the amount or magnitude of the difference in means (the bad), and its deficiency in that its outcome essentially depends completely on the sample size (the ugly). The t-test’s failure to provide the amount of difference offers no basis for us to judge practical significance. Furthermore, it makes metaanalysis impossible or ineffective and, therefore, seriously hinders the advancement of educational research. Its direct dependency on sample size makes a t-test virtually useless in providing reliable information. Effect size (ES) and confidence interval (CI) for ES are recommended to address t-test’s inadequacies by serving as a supplement to or replacement of the t-test. The commonly used effect size indices are reviewed and discussed in regard to their performance and robustness.