User-centered design of a mobile java practice app: A comparison of question formats

Abstract

Learning computer science presents many challenges to students, and providing resources for meaningful practice is recognized as a way to support rigorous learning and diverse student participation. At the same time, mobile phones are increasingly ubiquitous, creating an underutilized opportunity for practice outside of traditional methods. This experience report presents a user-centered approach to designing a practice app for introductory Java. We investigated user preferences through a series of small studies, first conducting think-aloud sessions and focus groups, and finally conducting a usability study comparing two prototype versions. The initial studies suggested how to leverage the affordances of small screens, ruling out free-response practice problems in favor of either fill-inthe- blank (FB) or multiple-choice (MC) questions. The comparison study revealed statistically significant differences in students' survey responses: (1) usability scores were significantly higher for the MC version than the FB version; (2) students reported significantly greater satisfaction and desire to learn for the MC version; and (3) students reported enjoying and being more comfortable with the MC version compared to the FB version. We contextualize this observation within related research on question formats. Takeaways from this experience report can provide guidance on designing mobile applications that give students opportunities for meaningful practice.