Does neatness count? What the organization of student work says about understanding

Abstract

Students have long been taught that neatness counts. But does it? In this project, we seek to understand how the organization of a student's solution to a problem relates to the correctness of the work. Understanding this relationship will enable us to create software to provide early warnings to students who may be struggling in a course. In this study, students in an undergraduate statics course completed all of their work, including homework, quizzes, and exams, using Livescribe™ Smartpens. These devices record the solutions as time-stamped pen strokes, enabling us to see not only the final ink on the page, but also the order in which it was written. Using this unique database of student work, we examine how the history of the solution construction process correlates with the correctness of the work. We characterize solution histories with a number of quantitative features describing the temporal and spatial organization of the work. For example, there are features that describe the order in which various problem solving activities, such as the construction of free body diagrams and equilibrium equations, are performed, and the amount of time spent on each activity. The spatial organization of the work is characterized by the extent to which a student revisits earlier parts of a solution to revise their work. Regression models have demonstrated that, on average, about 40% of the variance in student performance could be explained by our features. This is a surprising result in that the features consider only the process of recording the solution history and do not actually consider the semantics of the writing. © 2012 American Society for Engineering Education.