How the granularity of evaluation affects reliability of peer-assessment modelization in the OpenAnswer system

Abstract

The OpenAnswer system has the goal of exploiting teacher mediated peer-assessment for the evaluation of answers to open ended questions. The system models both the learning state of each student and their choices during peer-assessment. In OpenAnswer, each student is represented as a Bayesian network made of a triple of finite-domain variables: K for student's Knowledge about a topic, J for the estimated ability to evaluate ("Judge") the answer of another peer, C for Correctness of the answer to a given question. The student's individual sub-networks are connected through further Bayesian variables which model each peer-assessment choice, depending on the type of peer-assessment performed: (G for grading, B for choosing the best, W for choosing the worst). During an assessment session, each student grades a fixed number of peers' answers. The final result for a given session is a full set of grades for all students' answers, although the teacher had actually graded only a part of them. The student's assessments are instantiated in the network as evidence, together with the teacher's (perhaps partially complete) grades, so that OpenAnswer deduces the remaining grades. In the former OpenAnswer implementation, all variables were represented through a probability distribution over three values (Good/Fair/Bad for K and J, correct/fair/wrong for C). We present experiments and simulations showing that, by increasing the domain granularity for all variables from 3 to 6 values (A to F), the information obtained from the Bayesian network achieves higher reliability. © 2014 Springer International Publishing Switzerland.