Accuracy of a Classical Test Theory–Based Procedure for Estimating the Reliability of a Multistage Test

Abstract

The purpose of this simulation study was to assess the accuracy of a classical test theory ( CTT )–based procedure for estimating the alternate‐forms reliability of scores on a multistage test ( MST ) having 3 stages. We generated item difficulty and discrimination parameters for 10 parallel, nonoverlapping forms of the complete 3‐stage test and ability parameters for a population of 30,000 simulated test takers. Using these parameters to generate item responses, we ran each of the 30,000 simulated test takers through each of the 10 simulated forms of the 3‐stage test and computed the correlation of the scores on each pair of simulated test forms. We then computed the CTT estimate of the reliability, in the full population of 30,000 simulated test takers, of the total scaled scores resulting from the multistage testing procedure. We computed the estimate separately from the simulated responses to each of the 10 simulated forms of the 3‐stage test. The reliability estimates from each simulated form of the MST differed by less than .005 from the average of the correlations between scores on that form and on the other 9 simulated forms of the MST . Report Number: ETS RR‐17–02