Tips and tricks for building bayesian networks for scoring game-based assessments

Abstract

Game-based assessments produce multiple, dependent observations from student game play. Bayesian networks can model the dependence, but, typically, only a small amount of pilot data are available at the time the network is constructed. This paper examines the process of creating Bayesian network scoring models, focusing on several practical techniques that have been used in the construction of models for Physics Playground. In particular, the following techniques are helpful: (1) The use of evidence-centered assessment design to define latent competency variables and observable indicator variables. (2) The use of correlation matrixes to uncover and validate the conditional independence structure of the Bayes net. (3) The use of discrete IRT models to create large portion of the Bayesian networks from a single spreadsheet. (4) Adjusting the Bayes net parameters using both hand tuning and a generalized EM algorithm, creating networks which are a mixture of expert opinion and data. (5) Using expected classification accuracy matrixes to judge assessment validity and reliability. (6) Using evidence balance sheets to identify unusual subjects and observable indicators.