Evaluating multiple choice question generator

Abstract

Semantic-based computer-assisted automated question generator has been increasingly popular as a tool for creating personalized assessment questions.Various question generator tools have been proposed, such as those which generate structured question from a text file, generate Multiple Choice Question (MCQ) from a text file or from ontology-based knowledge representation. A comparison framework and evaluation methodology is required to evaluate different question generator tools. This paper discusses the requirement and criteria to carry out performance comparison of different MCQ generator. A feature comparison of Question Generation (QG) tool, namely Mimos-QG with the existing QG tools is presented. We have evaluated our QG tool based on several standard criteria such as the correctness of (a) distractor generation, (b) answer choice grouping strategy and (c) syntactical and pedagogical quality on three different domain ontologies. The experimental result indicated that Mimos-QG is capable of producing good quality direct type and grouping type multiple choice questions. © 2012 Springer-Verlag.