An Investigation of Reliability Coefficients Estimated for Decision Studies in Generalizability Theory

Abstract

This research aimed to compare the G and Phi coefficients estimated in Decision studies in Generalizability theory and obtained in actual cases for the same conditions of similar facets by using crossed design. The research was conducted as pure research on 120 individuals (students), six items and 12 raters. An achievement test composed of six open ended questions and a holistic rubric developed by the researcher were used in data collection. Data analysis included obtaining the G and Phi coefficients by creating actual cases for two, four and six raters followed by D studies conducted for other actual cases with different measurement conditions to make estimates. Finally, G and Phi coefficients obtained and estimated for two, four and six raters were compared separately. Findings show that G and Phi coefficients estimated in D studies by increasing the number of raters were sometimes greater than those obtained in actual cases although they were sometimes smaller. However, it was concluded that a pattern may not always exist between the G and Phi coefficients obtained in actual cases for same number of raters and those estimated in D studies.