Differences in interpretation of p values and effect size in Japanese journal of educational psychology

Abstract

In line with the movement for statistical reform in psychology, awareness of the limitations of null hypothesis significance testing (NHST) has increased among educational and psychological researchers. However, the actual discrepancy between the results from null hypothesis significance testing and alternative measures such as effect size has not been thoroughly examined in Japan. The present article reports the results from an exhaustive survey of articles published in the last 4 years in the Japanese Journal of Educational Psychology, and investigates the relation between published p values and effect size measures, using information derived from the summary and test statistics reported in those articles. The tests involved included independent group t-tests, paired t-tests, one-way ANOVA, and two-way factorial ANOVA. The results showed that, although moderate correlations were observed between p values and effect size measures, several of the studies examined were underpowered because the sample size was insufficient. On the other hand, several other studies reported very small effect sizes that were still significant in the context of null hypothesis significance testing. Future directions are discussed, including the use of meta-analyses and a Bayesian approach.