Does student crowdsourcing of practice questions and animations lead to good quality materials?

Abstract

Web-native textbooks use practice questions and animations to improve student performance to help learn and visualize concepts. Creating practice questions and animations is time intensive. This paper investigates whether having students create and/or rate practice questions and animations - student crowdsourcing - can lead to good quality items. For animations, we conducted experiments involving 587 participants from a basic computing technology course. Students were asked to create animations of a topic, and a professor and fellow students rated those animations. Some students showed the ability to create quality animations: 2 of 19 completed animations were rated a "5" on a 1 to 5 scale (5 is best) by a professor / professional animation author. Furthermore, some students showed the ability to effectively rate student-created animations; the top 10% of student ratings was strongly correlated with the professor ratings with R-value = 0.88 (p-value < 0.001). For questions, we conducted experiments involving 25 participants from an introductory embedded programming course. Students were asked to create and rate practice questions for various embedded programming topics. Some students could effectively rate questions: the average of the top 20% of student ratings was strongly correlated with the professor rating with R-value = 0.82 (p-value = 0.02). However, students did not show the ability to create good question; no student's question was professor-rated above a 4. The common problem seen was an inability to write correct and precise English.