How do machine-generated questions compare to human-generated questions?

Abstract

Science instructors need questions for use in exams, homework assignments, class discussions, reviews, and other instructional activities. Textbooks never have enough questions, so instructors must find them from other sources or generate their own questions. In order to supply biology instructors with questions for college students in introductory biology classes, two algorithms were developed. One generates questions from a formal representation of photosynthesis knowledge. The other collects biology questions from the web. The questions generated by these two methods were compared to questions from biology textbooks. Human students rated questions for their relevance, fluency, ambiguity, pedagogy, and depth. Questions were also rated by the authors according to the topic of the questions. Although the exact pattern of results depends on analytic assumptions, it appears that there is little difference in the pedagogical benefits of each class, but the questions generated from the knowledge base may be shallower than questions written by professionals. This suggests that all three types of questions may work equally well for helping students to learn.