Investigating Robustness of Item Response Theory Proficiency Estimators to Atypical Response Behaviors Under Two‐Stage Multistage Testing

Abstract

The purpose of this study is to evaluate the extent to which item response theory ( IRT ) proficiency estimation methods are robust to the presence of aberrant responses under the GRE ® General Test multistage adaptive testing ( MST ) design. To that end, a wide range of atypical response behaviors affecting as much as 10% of the test items were simulated using a generic GRE 2‐stage MST and the 2‐parameter logistic ( 2PL ) IRT model. As expected, some differences were found among the 5 estimators in terms of the recovery of the true theta ability; for example, Bayesian estimators had lower error variance and their estimates were regressed to the mean. Once the IRT theta estimates were scaled onto a comparable reporting score scale, however, it was found that all the estimation methods investigated, including the one currently used to score GRE MSTs , were equally robust under the simulated conditions. Report Number: ETS RR‐16–22