Examination of Testlet Effect in Open-Ended Items

Abstract

The testlets are prevalently used in national and international standard tests and the teacher-made achievement tests thanks to their practicality. The purpose of the current study is to evaluate the testlet effect in test consisting the open-ended items and to identify the possible errors when this effect is ignored. The data consists of the scores given by three raters of the responses of 380 students in the teacher training program to two open-ended items in two testlets. The two designs with and without the testlet effect were modeled (totally crossed random design and the nested random design) and the differences between them were analyzed by Generalizability Theory (GT). The results showed that raters had a slight effect on the scores in both models tested, and the reliability of the scores was overestimated when the testlet effect was not ignored in such measurement tools.