On-the-fly parameter estimation based on item response theory in item-based adaptive learning systems

Abstract

Online learning systems are able to offer customized content catered to individual learner’s needs, and have seen growing interest from industry and academia alike in recent years. In contrast to the traditional computerized adaptive testing setting, which has a well-calibrated item bank with new items added periodically, the online learning system has two unique features: (1) the number of items is large, and they have likely not gone through costly field testing for item calibration; and (2) the individual’s ability may change as a result of learning. The Elo rating system has been recognized as an effective method for fast updating of item and person parameters in online learning systems to enable personalized learning. However, the updating parameter in Elo has to be tuned post hoc, and Elo is only suitable for the Rasch model. In this paper, we propose the use of a moment-matching Bayesian update algorithm to estimate item and person parameters on the fly. With sequentially updated item and person parameters, a modified maximum posterior weighted information criterion (MPWI) is proposed to adaptively assign items to individuals. The Bayesian updated algorithm along with MPWI is validated in a simulated multiple-session online learning setting, and the results show that the new combo can achieve fast and reasonably accurate parameter estimations that are comparable to random selection, match-difficulty selection, and traditional online calibration. Moreover, the combo can still function reasonably well with as low as 20% of items being pre-calibrated in the item bank.