JIME http://jime.open.ac.uk/2008/14
1
The Open Learning Initiative: Measuring the Effectiveness of the OLI Statistics Course in Accelerating Student Learning Marsha Lovett, Oded Meyer, and Candace Thille
Carnegie Mellon University 5000 Forbes Ave. Pittsburgh USA www.cmu.edu


Abstract: The Open Learning Initiative (OLI) is an open educational resources project at Carnegie Mellon University that began in 2002 with a grant from The William and Flora Hewlett Foundation. OLI creates web-based courses that are designed so that students can learn effectively without an instructor. In addition, the courses are often used by instructors to support and complement face-to-face classroom instruction. Our evaluation efforts have investigated OLI courses’ effectiveness in both of these instructional modes – stand-alone and hybrid. This report documents several learning effectiveness studies that were focused on the OLI-Statistics course and conducted during Fall 2005, Spring 2006, and Spring 2007. During the Fall 2005 and Spring 2006 studies, we collected empirical data about the instructional effectiveness of the OLI-Statistics course in stand-alone mode, as compared to traditional instruction. In both of these studies, in-class exam scores showed no significant difference between students in the stand-alone OLI-Statistics course and students in the traditional instructor-led course. In contrast, during the Spring 2007 study, we explored an accelerated learning hypothesis, namely, that learners using the OLI course in hybrid mode will learn the same amount of material in a significantly shorter period of time with equal learning gains, as compared to students in traditional instruction. In this study, results showed that OLI-Statistics students learned a full semester’s worth of material in half as much time and performed as well or better than students learning from traditional instruction over a full semester. Keywords: Open Educational Resources, Evaluation, Online Courses, Learning Studies, Accelerated Learning, Interactive demonstration: A demonstration of the StatTutor statistics tutorial is available for playback from http://jime.open.ac.uk/2008/14/stattutor_tour/ . The demonstration is in Flash format. 1 Introduction The Open Learning Initiative (OLI) is an open educational resources project at Carnegie Mellon University that began in 2002 with a grant from The William and Flora Hewlett Foundation. Like many open educational resources projects, ours makes its courses openly and freely available. Our goal has been to create complete online courses that enact instruction: they offer structure, information, activities, practice, and feedback — all arranged so that students can learn even if they do not have the benefit of an instructor or classmates. Each of our courses is developed by a team composed of learning scientists, faculty content experts, human-computer interaction

JIME http://jime.open.ac.uk/2008/14
2
experts, and software engineers in order to make best use of multidisciplinary knowledge for designing effective instruction. Moreover, as students work through the OLI courses, we collect real-time, interaction-level data on how they are learning, and we use this data to inform further course revisions and improvements. In addition to this ongoing formative evaluation, we conduct formal learning studies on a regular basis. The studies reported here investigated the effectiveness of the OLI-Statistics course by comparison to traditional instruction. The overall goal was not to contrast online versus face-to-face delivery of instruction but rather to test whether the learning experience offered through the OLI-Statistics course was comparable to (or better than) that afforded by traditional instruction so that (a) the effectiveness of the OLI design could be validated for this particular course and (b) students who, for whatever reason, do not have access to a full-semester course in undergraduate Statistics could be assured of an equivalently effective alternative in the form of OLI-Statistics. More specifically, the primary goal of the first two studies was to test the hypothesis that students would learn as much from the OLI-Statistics course in stand-alone mode as they would from traditional, instructor-led instruction. This goal represents a fairly simplified “do no harm” test of the stand-alone version of OLI-Statistics (i.e., students’ learning would not be harmed relative to taking Statistics in a traditional face-to-face setting). The primary goal of the third study was to test the hypothesis that students using the OLI-statistics course in hybrid mode (i.e., online learning combined with classroom instruction) could learn a semester’s worth of material in half the time and yet to the same level of mastery as students learning from traditional instruction. This “accelerated learning” test involved a more rigorous evaluation of the hybrid version of OLI-Statistics compared to a fully instructor-led Statistics course and used the more sensitive measure of learning efficiency (i.e., amount learned per unit time) instead of total learning gain. The secondary goal of all three studies was to investigate students’ patterns of use of the OLI materials (and any correlations with their learning outcomes) in order to inform further development and refinement of the course. We should also note that, although all of the studies reported here were conducted with students from Carnegie Mellon, our next study – currently ongoing – seeks to extend the generalizability of the present results by conducting a similar investigation with community college students. The following sections of this report discuss, in turn, the design of the OLI-Statistics course, the two preliminary “do no harm” studies we conducted (including their research design, student-learning measures, and basic results), the third “accelerated learning” study (including its research design, student-learning measures, basic results, and a follow-up retention study), and a general interpretation of our results in light of learning theory and in terms of potential uses for the OLI-Statistics course. While this report presents multiple analyses of the data collected, continuing analysis efforts are ongoing. 2 Description and Design of the OLI Statistics Course The OLI-Statistics course was designed to teach the same material as covered in the Introductory Statistics course taught face-to-face at Carnegie Mellon. That course represents a typical college-level, non-calculus-based introduction to statistics, so the content for OLI-Statistics course was well established. In contrast, the format and activities incorporated in the OLI-Statistics course were newly designed to incorporate several additional sources of information: the experience and knowledge of statistics faculty members involved in the course development, specific research findings regarding how students learn statistics, and more general empirical and theoretical results from research in the learning sciences. The subsections below illustrate several design features of the course, highlighting differences from the face-to-face course.