Exploring a new approach to the assessment of web-based materials for engineering statics course

Abstract

The OLI Engineering Statics course offers web-based instruction that provides many embedded interactive exercises offering individualized feedback and help to students, while giving instructors summary information on student performance. Various efforts to assess the effectiveness of courseware have been presented in the past and summarized here. As demonstrated in past research, students exhibit significant gains in diagnostic quizzes after using OLI modules. Also, quiz scores have been shown to correlate positively with other important measures of performance, such as exams and statics concept inventory scores. However, it has been found that overall course performances do no correlate with how many optional exercises a student initiates. This result was to some extent expected, since in these studies students were not required to undertake a certain number of exercises, but rather to do as many as they deemed necessary to perform well on the end-of-module quiz. Student usage was indeed found to vary widely, indicating self-regulation, which lessens the likelihood of a positive correlation between the number of exercises undertaken and performance. As an alternative to overall performance in a course, one can look at a much smaller grain size and potentially detect whether repeated exercises on specific concepts and skills do lead to fewer errors. With the goal assessing the benefit of engaging in specific exercises, a new approach has been presented here, which adapts the method of Learning Curve Analysis that is common in intelligent tutoring systems. Each individual question within each exercise is tagged as corresponding to one or more skills. Then, successive opportunities for students to exercise each skill are tracked. If feedback on each opportunity is effective, then the students should make fewer errors in successive opportunities to practice a given skill. It has been found that for some skills there is clearly improvement with practice. For other skills, the variation with practice is much less clear. Various reasons for the absence of improvement were proposed. Interactive exercises often require multiple skills, which are harder to disentangle, sometimes there might be insufficient opportunities to practice a skill, and sometimes different opportunities involve applications of the same skill that are arguably quite distinct in difficulty. Methods for gauging the effectiveness of web-based materials should continue to be sought, including comparisons with alternative instruction. Yet, when overall performance is taken to be the basis for comparison it is difficult to pinpoint where specifically the materials could be improved. Learning Curve Analysis offers a potentially valuable means of determining whether students are making progress in acquiring specific skills addressed by a collection of exercises, and thereby point to particular exercises where improvements should be sought.