Guidelines of the minimum sample size requirements for Cohen’s Kappa

Abstract

Background: To estimate sample size for Cohen’s kappa agreement test can be challenging especially when dealing with various effect sizes. This study aimed to present minimum sample size determination for Cohen’s kappa under different scenarios when certain assumptions are held. Methods: The sample size formula was introduced by Flack and colleagues (1988). The power was pre-specified to be at least 80% and 90% while alpha was set at less than 0.05. The effect sizes were derived from several prespecified estimates such as the pattern of the true marginal rating frequencies and the difference between the two kappa coefficients in the hypothesis testing. Results: When the true marginal rating frequencies are the same, the minimum sample size determination ranges from 2 to 927 depending on the actual value of the effect size. When the true marginal rating frequencies are not the same, then the majority of the minimum sample size required for this condition is more than double than that required sample size when the true marginal rating frequencies are the same. Conclusion: Concerning that the sample size formula could produce a very extreme small sample size, the determination of K1 and K2 should be based on reasonable estimates. We recommend for all sample size determinations for Cohen’s kappa agreement test, the true marginal rating frequencies can be assumed to be the same. Otherwise, it will be necessary to multiply the estimated minimum sample size by two to accommodate if the true marginal rating frequencies are not the same.