Constrained Adaptive Testing with Shadow Tests

Abstract

Though adaptive testing research was initially motivated by the in- tention to make testing statistically more informative, the first real-life testing programs to make the transition to CAT quickly discovered that adaptive testing operating only on this principle would lead to unre- alistic results. For example, if items are selected only to maximize the information in the ability estimator, test content may easily become un- balanced for some ability levels. If examinees happened to learn about this feature, they may change their test preparations and the item cali- bration results would no longer be valid. Likewise, without any further provisions, adaptive tests with maximum information can also be un- balanced with respect to such attributes as their possible orientation towards gender or minority groups and become unfair for certain groups of examinees. Furthermore, even a simple attribute such as the serial position of the correct answer for the items could become a problem if the adaptive test administrations produced highly disproportionate use of particular answer keys. Lower ability examinees might benefit from patterned guessing and some of the more able examinees might become anxious and begin second-guessing their answers to previous items. Examinees may get alerted by this fact and start bothering if their answers to the previous questions were correct. More examples of nonstatistical specifications for adaptive tests are easy to provide. In fact, what most testing programs want if they make the transition from linear to adaptive testing, is test administrations that have exactly the same “look and feel” as their old linear forms but that are much shorter because of a better adaptation to the ability levels of the individual examinees. The point is that adaptive testing will only be accepted if the statistical principle of adapting the item selections to the ability estimates for examinees is implemented in conjunction with serious consideration of many other nonstatistical test specifications. Formally, each specification that an adaptive test has to meet im- poses a constraint on the selection of the items from the pool. As a con- sequence, a CAT algorithm that combines maximization of statistical information with the realization of several nonstatistical specifications can be viewed as an algorithm for constrained sequential optimization. The objective function to be optimized is the statistical information in the test items at the current ability estimate. All other specifications are the constraints subject to which the optimization has to take place. The goal of this chapter is to develop this point of view further and present a general method of constrained sequential optimization for application in adaptive testing. This method has proven to be successful in several applications. The basic principle underlying the method is to implement all constraints through a series of shadow tests assembled to be optimal at the updated ability estimates of the examinee. The items to be administered are selected from these shadow tests rather than directly from the item pool. Use of the method will be illustrated for item pools from several well-known large-scale testing programs.