AN ADAPTIVE ALGEBRA TEST: A TESTLET‐BASED, HIERARCHICALLY‐STRUCTURED TEST WITH VALIDITY‐BASED SCORING§

Abstract

Earlier (Wainer & Lewis, 1990) we reported the initial development of a testlet‐based algebra test. In this account we provide the details of this excursion into the use of hierarchical testlets and validity‐based scoring. A pretest of two 15 item hierarchical testlets was carried out in which examinees' performance on a four item subset of each testlet was used to predict performance on the entire testlet. Four models for constructing hierarchies were considered. These presentation hierarchies were compared with one another and with an optimally chosen set of four linearly administered items. The comparison was carried out using both the root mean square error and the conditional posterior variance as the criterion. It was found on cross validation that although an adaptive test is everywhere superior to a fixed format test, this superiority is crucially dependent upon the quality of the items. When items vary considerably in quality a fixed format test, which uses the best items, can do almost as well as an adaptive test of equal length.