Evaluation of Evidence-Based Practices in
Online Learning: A Meta-Analysis and
Review of Online Learning Studies





U.S. Department of Education
Office of Planning, Evaluation, and Policy Development
Policy and Program Studies Service








Prepared by
Barbara Means
Yukie Toyama
Robert Murphy
Marianne Bakia
Karla Jones
Center for Technology in Learning

2009
Файл загружен с http://www.ifap.ru

This report was prepared for the U.S. Department of Education under Contract number ED-04-
CO-0040 Task 0006 with SRI International. Bernadette Adams Yates served as the project
manager. The views expressed herein do not necessarily represent the positions or policies of the
Department of Education. No official endorsement by the U.S. Department of Education is
intended or should be inferred.

U.S. Department of Education
Arne Duncan
Secretary
Office of Planning, Evaluation and Policy Development
Carmel Martin
Assistant Secretary
Policy and Program Studies Service Office of Educational Technology
Alan Ginsburg Laura Johns
Director Acting Director
Program and Analytic Studies Division
David Goodwin
Director

May 2009

This report is in the public domain. Authorization to reproduce this report in whole or in part is granted.
Although permission to reprint this publication is not necessary, the suggested citation is: U.S.
Department of Education, Office of Planning, Evaluation, and Policy Development, Evaluation of
Evidence-Based Practices in Online Learning: A Meta-Analysis and Review of Online Learning Studies,
Washington, D.C., 2009.
This report is also available on the Department’s Web site at
www.ed.gov/about/offices/list/opepd/ppss/reports.html.
On request, this publication is available in alternate formats, such as braille, large print, or computer
diskette. For more information, please contact the Department’s Alternate Format Center at
(202) 260-0852 or (202) 260-0818.

Contents

EXHIBITS ...................................................................................................................................................................... V
ACKNOWLEDGMENTS ................................................................................................................................................ VII
ABSTRACT ................................................................................................................................................................... IX
EXECUTIVE SUMMARY ............................................................................................................................................... XI
Literature Search .................................................................................................................................................... xii
Meta-Analysis ....................................................................................................................................................... xiii
Narrative Synthesis ................................................................................................................................................ xiv
Key Findings .......................................................................................................................................................... xiv
Conclusions ............................................................................................................................................................ xvi
1. INTRODUCTION ......................................................................................................................................................... 1
Context for the Meta-analysis and Literature Review .............................................................................................. 2
Conceptual Framework for Online Learning ............................................................................................................ 3
Findings From Prior Meta-Analyses ......................................................................................................................... 6
Structure of the Report .............................................................................................................................................. 7
2. METHODOLOGY ........................................................................................................................................................ 9
Definition of Online Learning .................................................................................................................................. 9
Data Sources and Search Strategies ........................................................................................................................ 10
Electronic Database Searches ................................................................................................................................. 10
Additional Search Activities ................................................................................................................................... 10
Screening Process ................................................................................................................................................... 11
Effect Size Extraction ............................................................................................................................................. 13
Coding of Study Features ....................................................................................................................................... 14
Data Analysis .......................................................................................................................................................... 15
3. FINDINGS ................................................................................................................................................................. 17
Nature of the Studies in the Meta-Analysis ............................................................................................................ 17
Main Effects ............................................................................................................................................................ 18
Test for Homogeneity ............................................................................................................................................. 27
Analyses of Moderator Variables ........................................................................................................................... 27
Practice Variables ................................................................................................................................................... 28
Condition Variables ................................................................................................................................................ 30
Methods Variables .................................................................................................................................................. 31
4. NARRATIVE SYNTHESIS OF STUDIES COMPARING VARIANTS OF ONLINE LEARNING ........................................ 37
Blended Compared With Pure Online Learning ..................................................................................................... 38
Media Elements ...................................................................................................................................................... 40
Learning Experience Type ...................................................................................................................................... 41
Computer-Based Instruction ................................................................................................................................... 43
Supports for Learner Reflection .............................................................................................................................. 44
Moderating Online Groups ..................................................................................................................................... 46
Scripts for Online Interaction .................................................................................................................................. 46
Delivery Platform ................................................................................................................................................... 47
Summary ................................................................................................................................................................. 48
5. DISCUSSION AND IMPLICATIONS ............................................................................................................................ 51
Comparison With Meta-Analyses of Distance Learning ........................................................................................ 52
Implications for K–12 Education ............................................................................................................................ 53
iii

REFERENCES ............................................................................................................................................................... 55
Reference Key ........................................................................................................................................................ 55
APPENDIX META-ANALYSIS METHODOLOGY ....................................................................................................... A-1
Terms and Processes Used in the Database Searches .......................................................................................... A-1
Additional Sources of Articles ............................................................................................................................. A-3
Effect Size Extraction .......................................................................................................................................... A-4
Coding of Study Features .................................................................................................................................... A-5
iv

v
Exhibits
Exhibit 1. Conceptual Framework for Online Learning ................................................................................................ 5
Exhibit 2. Bases for Excluding Studies During the Full-Text Screening Process ....................................................... 13
Exhibit 3. Effect Sizes for Contrasts in the Meta-Analysis ......................................................................................... 20
Exhibit 4a. Purely Online Versus Face-to-Face (Category 1) Studies Included in the Meta-Analysis ........................ 21
Exhibit 4b. Blended Versus Face-to-Face (Category 2) Studies Included in the Meta-Analysis ................................ 24
Exhibit 5. Tests of Practices as Moderator Variables .................................................................................................. 29
Exhibit 6. Tests of Conditions as Moderator Variables ............................................................................................... 30
Exhibit 7. Studies of Online Learning Involving K–12 Students ................................................................................ 32
Exhibit 8. Tests of Study Features as Moderator Variables ......................................................................................... 34
Exhibit 9. Learner Types for Category 3 Studies ........................................................................................................ 37
Exhibit A-1. Terms for Initial Research Database Search ........................................................................................ A-2
Exhibit A-2. Terms for Additional Database Searches for Online Career Technical Education and Teacher
Professional Development ............................................................................................................................... A-2
Exhibit A-3. Sources for Articles in the Full-Text Screening ................................................................................... A-3
Exhibit A-4. Top-level Coding Structure for the Meta-analysis ............................................................................... A-6

Acknowledgments

We would like to acknowledge the thoughtful contributions of the members of our Technical
Work Group in reviewing study materials and prioritizing issues to investigate. The advisors
consisted of Robert M. Bernard of Concordia University, Richard E. Clark of the University of
Southern California, Barry Fishman of the University of Michigan, Dexter Fletcher of the
Institute for Defense Analysis, Karen Johnson of the Minnesota Department of Education, Mary
Kadera of PBS, James L. Morrison an independent consultant, Susan Patrick of the North
American Council for Online Learning, Kurt D. Squire of the University of Wisconsin, Bill
Thomas of the Southern Regional Education Board, Bob Tinker of The Concord Consortium,
and Julie Young of the Florida Virtual School. Robert M. Bernard, the Technical Work Group’s
meta-analysis expert, deserves a special thanks for his advice and sharing of unpublished work
on meta-analysis methodology as well as his careful review of an earlier version of this report.
Many U.S. Department of Education staff members contributed to the completion of this report.
Bernadette Adams Yates served as project manager and provided valuable substantive guidance
and support throughout the design, implementation and reporting phases of this study. We would
also like to acknowledge the assistance of other Department staff members in reviewing this
report and providing useful comments and suggestions, including David Goodwin, Sue Betka
and Mike Smith.
We appreciate the assistance and support of all of the above individuals; any errors in judgment
or fact are of course the responsibility of the authors.
The Evaluation of Evidence-Based Practices in Online Learning is supported by a large project
team at SRI International. Among the staff members who contributed to the research were Sarah
Bardack, Ruchi Bhanot, Kate Borelli, Sara Carriere, Katherine Ferguson, Reina Fujii, Joanne
Hawkins, Ann House, Katie Kaattari, Klaus Krause, Yessica Lopez, Lucy Ludwig, Patrik Lundh,
L. Nguyen, Julie Remold, Elizabeth Rivera, Luisana Sahagun Velasco, Mark Schlager, and Edith
Yang.

vii

Abstract

A systematic search of the research literature from 1996 through July 2008 identified more than
a thousand empirical studies of online learning. Analysts screened these studies to find those that
(a) contrasted an online to a face-to-face condition, (b) measured student learning outcomes, (c)
used a rigorous research design, and (d) provided adequate information to calculate an effect
size. As a result of this screening, 51 independent effects were identified that could be subjected
to meta-analysis. The meta-analysis found that, on average, students in online learning
conditions performed better than those receiving face-to-face instruction. The difference
between student outcomes for online and face-to-face classes—measured as the difference
between treatment and control means, divided by the pooled standard deviation—was larger in
those studies contrasting conditions that blended elements of online and face-to-face instruction
with conditions taught entirely face-to-face. Analysts noted that these blended conditions often
included additional learning time and instructional elements not received by students in control
conditions. This finding suggests that the positive effects associated with blended learning
should not be attributed to the media, per se. An unexpected finding was the small number of
rigorous published studies contrasting online and face-to-face learning conditions for K–12
students. In light of this small corpus, caution is required in generalizing to the K–12 population
because the results are derived for the most part from studies in other settings (e.g., medical
training, higher education).
ix

Executive Summary
Online learning—for students and for teachers—is one of the fastest growing trends in
educational uses of technology. The National Center for Education Statistics (2008) estimated
that the number of K-12 public school students enrolling in a technology-based distance
education course grew by 65 percent in the two years from 2002-03 to 2004-05. On the basis of a
more recent district survey, Picciano and Seaman (2009) estimated that more than a million K–
12 students took online courses in school year 2007–08.
Online learning overlaps with the broader category of distance learning, which encompasses
earlier technologies such as correspondence courses, educational television and
videoconferencing. Earlier studies of distance learning concluded that these technologies were
not significantly different from regular classroom learning in terms of effectiveness. Policy-
makers reasoned that if online instruction is no worse than traditional instruction in terms of
student outcomes, then online education initiatives could be justified on the basis of cost
efficiency or need to provide access to learners in settings where face-to-face instruction is not
feasible. The question of the relative efficacy of online and face-to-face instruction needs to be
revisited, however, in light of today’s online learning applications, which can take advantage of a
wide range of Web resources, including not only multimedia but also Web-based applications
and new collaboration technologies. These forms of online learning are a far cry from the
televised broadcasts and videoconferencing that characterized earlier generations of distance
education. Moreover, interest in hybrid approaches that blend in-class and online activities is
increasing. Policy-makers and practitioners want to know about the effectiveness of Internet-
based, interactive online learning approaches and need information about the conditions under
which online learning is effective.
The findings presented here are derived from (a) a systematic search for empirical studies of the
effectiveness of online learning and (b) a meta-analysis of those studies from which effect sizes
that contrasted online and face-to-face instruction could be extracted or estimated. A narrative
summary of studies comparing different forms of online learning is also provided.
These activities were undertaken to address four research questions:
1. How does the effectiveness of online learning compare with that of face-to-face
instruction?
2. Does supplementing face-to-face instruction with online instruction enhance learning?
3. What practices are associated with more effective online learning?
4. What conditions influence the effectiveness of online learning?
This meta-analysis and review of empirical online learning research are part of a broader study
of practices in online learning being conducted by SRI International for the Policy and Program
Studies Service of the U.S. Department of Education. The goal of the study as a whole is to
provide policy-makers, administrators and educators with research-based guidance about how to
implement online learning for K–12 education and teacher preparation. An unexpected finding of
the literature search, however, was the small number of published studies contrasting online and
xi

face-to-face learning conditions for K–12 students. Because the search encompassed the research
literature not only on K–12 education but also on career technology, medical and higher
education, as well as corporate and military training, it yielded enough studies with older learners
to justify a quantitative meta-analysis. Thus, analytic findings with implications for K–12
learning are reported here, but caution is required in generalizing to the K–12 population because
the results are derived for the most part from studies in other settings (e.g., medical training,
higher education).
This literature review and meta-analysis differ from recent meta-analyses of distance learning in
that they
• Limit the search to studies of Web-based instruction (i.e., eliminating studies of video-
and audio-based telecourses or stand-alone, computer-based instruction);
• Include only studies with random-assignment or controlled quasi-experimental designs;
and
• Examine effects only for objective measures of student learning (e.g., discarding effects
for student or teacher perceptions of learning or course quality, student affect, etc.).
This analysis and review distinguish between instruction that is offered entirely online and
instruction that combines online and face-to-face elements. The first of the alternatives to
classroom-based instruction, entirely online instruction, is attractive on the basis of cost and
convenience as long as it is as effective as classroom instruction. The second alternative, which
the online learning field generally refers to as blended or hybrid learning, needs to be more
effective than conventional face-to-face instruction to justify the additional time and costs it
entails. Because the evaluation criteria for the two types of learning differ, this meta-analysis
presents separate estimates of mean effect size for the two subsets of studies.
Literature Search
The most unexpected finding was that an extensive initial search of the published literature from
1996 through 2006 found no experimental or controlled quasi-experimental studies that both
compared the learning effectiveness of online and face-to-face instruction for K–12 students and
provided sufficient data for inclusion in a meta-analysis. A subsequent search extended the time
frame for studies through July 2008.
The computerized searches of online databases and citations in prior meta-analyses of distance
learning as well as a manual search of the last three years of key journals returned 1,132
abstracts. In two stages of screening of the abstracts and full texts of the articles, 176 online
learning research studies published between 1996 and 2008 were identified that used an
experimental or quasi-experimental design and objectively measured student learning outcomes.
Of these 176 studies, 99 had at least one contrast between an included online or blended learning
condition and face-to-face (offline) instruction that potentially could be used in the quantitative
meta-analysis. Just nine of these 99 involved K–12 learners. The 77 studies without a face-to-
face condition compared different variations of online learning (without a face-to-face control
condition) and were set aside for narrative synthesis.
xii

Meta-Analysis
Meta-analysis is a technique for combining the results of multiple experiments or quasi-
experiments to obtain a composite estimate of the size of the effect. The result of each
experiment is expressed as an effect size, which is the difference between the mean for the
treatment group and the mean for the control group, divided by the pooled standard deviation. Of
the 99 studies comparing online and face-to-face conditions, 46 provided sufficient data to
compute or estimate 51 independent effect sizes (some studies included more than one effect).
Four of the nine studies involving K–12 learners were excluded from the meta-analysis: Two
were quasi-experiments without statistical control for preexisting group differences; the other
two failed to provide sufficient information to support computation of an effect size.
Most of the articles containing the 51 effects in the meta-analysis were published in 2004 or
more recently. The split between studies of purely online learning and those contrasting blended
online/face-to-face conditions against face-to-face instruction was fairly even, with 28 effects in
the first category and 23 in the second. The 51 estimated effect sizes included seven contrasts
from five studies conducted with K–12 learners—two from eighth-grade students in social
studies classes, one for eighth- and ninth-grade students taking Algebra I, two from a study of
middle school students taking Spanish, one for fifth-grade students in science classes in Taiwan,
and one from elementary-age students in special education classes. The types of learners in the
remaining studies were about evenly split between college or community college students and
graduate students or adults receiving professional training. All but two of the studies involved
formal instruction. The most common subject matter was medicine or health care. Other content
types were computer science, teacher education, mathematics, languages, science, social science,
and business. Among the 49 contrasts from studies that indicated the time period over which
instruction occurred, 19 involved instructional time frames of less than a month, and the
remainder involved longer periods. In terms of instructional features, the online learning
conditions in these studies were less likely to be instructor-directed (8 contrasts) than they were
to be student-directed, independent learning (17 contrasts) or interactive and collaborative in
nature (23 contrasts).
Effect sizes were computed or estimated for this final set of 51 contrasts. Among the 51
individual study effects, 11 were significantly positive, favoring the online or blended learning
condition. Two contrasts found a statistically significant effect favoring the traditional face-to-
face condition.1

1 When a α < .05 level of significance is used for contrasts, one would expect approximately 1 in 20 contrasts to
show a significant difference by chance. For 51 contrasts, then, one would expect 2 or 3 significant differences by
chance. The finding of 2 significant contrasts associated with face-to-face instruction is clearly within the range
one would expect by chance; the 11contrasts associated with online or hybrid instruction exceeds what one would
expect by chance.
xiii

• Instruction combining online and face-to-face elements had a larger advantage relative
to purely face-to-face instruction than did purely online instruction. The mean effect size
in studies comparing blended with face-to-face instruction was +0.35, p < .001. This
effect size is larger than that for studies comparing purely online and purely face-to-face
conditions, which had an average effect size of +0.14, p < .05. An important issue to keep
in mind in reviewing these findings is that many studies did not attempt to equate (a) all
the curriculum materials, (b) aspects of pedagogy and (c) learning time in the treatment
and control conditions. Indeed, some authors asserted that it would be impossible to have
done so. Hence, the observed advantage for online learning in general, and blended
learning conditions in particular, is not necessarily rooted in the media used per se and
may reflect differences in content, pedagogy and learning time.
• Studies in which learners in the online condition spent more time on task than students in
the face-to-face condition found a greater benefit for online learning.5 The mean effect
size for studies with more time spent by online learners was +0.46 compared with +0.19
for studies in which the learners in the face-to-face condition spent as much time or more
on task (Q = 3.88, p < .05).6
• Most of the variations in the way in which different studies implemented online learning
did not affect student learning outcomes significantly. Analysts examined 13 online
learning practices as potential sources of variation in the effectiveness of online learning
compared with face-to-face instruction. Of those variables, (a) the use of a blended rather
than a purely online approach and (b) the expansion of time on task for online learners
were the only statistically significant influences on effectiveness. The other 11 online
learning practice variables that were analyzed did not affect student learning
significantly. However, the relatively small number of studies contrasting learning
outcomes for online and face-to-face instruction that included information about any
specific aspect of implementation impeded efforts to identify online instructional
practices that affect learning outcomes.
• The effectiveness of online learning approaches appears quite broad across different
content and learner types. Online learning appeared to be an effective option for both
undergraduates (mean effect of +0.35, p < .001) and for graduate students and
professionals (+0.17, p < .05) in a wide range of academic and professional studies.
Though positive, the mean effect size is not significant for the seven contrasts involving
K–12 students, but the number of K–12 studies is too small to warrant much confidence
in the mean effect estimate for this learner group. Three of the K–12 studies had
significant effects favoring a blended learning condition, one had a significant negative
effect favoring face-to-face instruction, and three contrasts did not attain statistical
significance. The test for learner type as a moderator variable was nonsignificant. No

5 This contrast falls just short of statistical significance (p < .06) when the five K-12 contrasts are removed from the
analysis.
6 The QBetween statistic tests whether the variances for the two sets of effect sizes under comparison are statistically
different.
xv

significant differences in effectiveness were found that related to the subject of
instruction.
• Effect sizes were larger for studies in which the online and face-to-face conditions varied
in terms of curriculum materials and aspects of instructional approach in addition to the
medium of instruction. Analysts examined the characteristics of the studies in the meta-
analysis to ascertain whether features of the studies’ methodologies could account for
obtained effects. Six methodological variables were tested as potential moderators: (a)
sample size, (b) type of knowledge tested, (c) strength of study design, (d) unit of
assignment to condition, (e) instructor equivalence across conditions, and (f) equivalence
of curriculum and instructional approach across conditions. Only equivalence of
curriculum and instruction emerged as a significant moderator variable (Q = 5.40, p <
.05). Studies in which analysts judged the curriculum and instruction to be identical or
almost identical in online and face-to-face conditions had smaller effects than those
studies where the two conditions varied in terms of multiple aspects of instruction (+0.20
compared with +0.42, respectively). Instruction could differ in terms of the way activities
were organized (for example as group work in one condition and independent work in
another) or in the inclusion of instructional resources (such as a simulation or instructor
lectures) in one condition but not the other.
The narrative review of experimental and quasi-experimental studies contrasting different online
learning practices found that the majority of available studies suggest the following:
• Blended and purely online learning conditions implemented within a single study
generally result in similar student learning outcomes. When a study contrasts blended
and purely online conditions, student learning is usually comparable across the two
conditions.
• Elements such as video or online quizzes do not appear to influence the amount that
students learn in online classes. The research does not support the use of some frequently
recommended online learning practices. Inclusion of more media in an online application
does not appear to enhance learning. The practice of providing online quizzes does not
seem to be more effective than other tactics such as assigning homework.
• Online learning can be enhanced by giving learners control of their interactions with
media and prompting learner reflection. Studies indicate that manipulations that trigger
learner activity or learner reflection and self-monitoring of understanding are effective
when students pursue online learning as individuals.
• Providing guidance for learning for groups of students appears less successful than does
using such mechanisms with individual learners. When groups of students are learning
together online, support mechanisms such as guiding questions generally influence the
way students interact, but not the amount they learn.
Conclusions
xvi

xvii
In recent experimental and quasi-experimental studies contrasting blends of online and face-to-
face instruction with conventional face-to-face classes, blended instruction has been more
effective, providing a rationale for the effort required to design and implement blended
approaches. Even when used by itself, online learning appears to offer a modest advantage over
conventional classroom instruction.
However, several caveats are in order: Despite what appears to be strong support for online
learning applications, the studies in this meta-analysis do not demonstrate that online learning is
superior as a medium, In many of the studies showing an advantage for online learning, the
online and classroom conditions differed in terms of time spent, curriculum and pedagogy. It was
the combination of elements in the treatment conditions (which was likely to have included
additional learning time and materials as well as additional opportunities for collaboration) that
produced the observed learning advantages. At the same time, one should note that online
learning is much more conducive to the expansion of learning time than is face-to-face
instruction.
In addition, although the types of research designs used by the studies in the meta-analysis were
strong (i.e., experimental or controlled quasi-experimental), many of the studies suffered from
weaknesses such as small sample sizes; failure to report retention rates for students in the
conditions being contrasted; and, in many cases, potential bias stemming from the authors’ dual
roles as experimenters and instructors.
Finally, the great majority of estimated effect sizes in the meta-analysis are for undergraduate
and older students, not elementary or secondary learners. Although this meta-analysis did not
find a significant effect by learner type, when learners’ age groups are considered separately, the
mean effect size is significantly positive for undergraduate and other older learners but not for
K–12 students.
Another consideration is that various online learning implementation practices may have
differing effectiveness for K–12 learners than they do for older students. It is certainly possible
that younger students could benefit more from a different degree of teacher or computer-based
guidance than would college students and older learners. Without new random assignment or
controlled quasi-experimental studies of the effects of online learning options for K–12 students,
policy-makers will lack scientific evidence of the effectiveness of these emerging alternatives to
face-to-face instruction.

1. Introduction
Online learning has roots in the tradition of distance education, which goes back at least 100
years to the early correspondence courses. With the advent of the Internet and the World Wide
Web, the potential for reaching learners around the world increased greatly, and today’s online
learning offers rich educational resources in multiple media and the capability to support both
real-time and asynchronous communication between instructors and learners as well as among
different learners. Institutions of higher education and corporate training were quick to adopt
online learning. Although K–12 school systems lagged behind at first, this sector’s adoption of e-
learning is now proceeding rapidly.
The National Center for Education Statistics estimated that 37 percent of school districts had
students taking technology-supported distance education courses during school year 2004–05
(Zandberg and Lewis 2008). Enrollments in these courses (which included two-way interactive
video as well as Internet-based courses), were estimated at 506,950, a 60 percent increase over
the estimate based on the previous survey for 2002-03 (Selzer and Lewis 2007). Two district
surveys commissioned by the Sloan Consortium (Picciano and Seaman 2007; 2008) produced
estimates that 700,000 K–12 public school students took online courses in 2005–06 and over a
million students did so in 2007–08—a 43 percent increase.7 Most of these courses were at the
high school level or in combination elementary-secondary schools (Zandberg and Lewis 2008).
These district numbers, however, do not fully capture the popularity of programs that are entirely
online. By fall 2007, 28 states had online virtual high school programs (Tucker 2007). The
largest of these, the Florida Virtual School, served over 60,000 students in 2007–08. In addition,
enrollment figures for courses or high school programs that are entirely online reflect just one
part of overall K–12 online learning. Increasingly, regular classroom teachers are incorporating
online teaching and learning activities into their instruction.
Online learning has become popular because of its potential for providing more flexible access to
content and instruction at any time, from any place. Frequently, the focus entails (a) increasing
the availability of learning experiences for learners who cannot or choose not to attend traditional
face-to-face offerings, (b) assembling and disseminating instructional content more cost-
efficiently, or (c) enabling instructors to handle more students while maintaining learning
outcome quality that is equivalent to that of comparable face-to-face instruction.
Different technology applications are used to support different models of online learning. One
class of online learning models uses asynchronous communication tools (e.g., e-mail, threaded
discussion boards, newsgroups) to allow users to contribute at their convenience. Synchronous
technologies (e.g., webcasting, chat rooms, desktop audio/video technology) are used to
approximate face-to-face teaching strategies such as delivering lectures and holding meetings
with groups of students. Earlier online programs tended to implement one model or the other.
More recent applications tend to combine multiple forms of synchronous and asynchronous
online interactions as well as occasional face-to-face interactions.

7 The Sloan Foundation surveys had very low response rates, suggesting the need for caution with respect to their
numerical estimates.
1

In addition, online learning offerings are being designed to enhance the quality of learning
experiences and outcomes. One common conjecture is that learning a complex body of
knowledge effectively requires a community of learners (Bransford, Brown and Cocking 1999;
Riel and Polin 2004; Schwen and Hara 2004; Vrasidas and Glass 2004) and that online
technologies can be used to expand and support such communities. Another conjecture is that
asynchronous discourse is inherently self-reflective and therefore more conducive to deep
learning than is synchronous discourse (Harlen and Doubler 2004; Hiltz and Goldman 2005;
Jaffee et al. 2006).
This literature review and meta-analysis have been guided by four research questions:
1. How does the effectiveness of online learning compare with that of face-to-face
instruction?
2. Does supplementing face-to-face instruction with online instruction enhance learning?
3. What practices are associated with more effective online learning?
4. What conditions influence the effectiveness of online learning?
Context for the Meta-analysis and Literature Review
The meta-analysis and literature review reported here are part of the broader Evaluation of
Evidence-Based Practices in Online Learning study that SRI International is conducting for the
Policy and Program Studies Service of the U.S. Department of Education. The overall goal of the
study is to provide research-based guidance to policy-makers, administrators and educators for
implementing online learning for K–12 education. This literature search, analysis, and review
has expanded the set of studies available for analysis by also addressing the literature concerning
online learning in career technical education, medical and higher education, corporate and
military training, and K–12 education.
In addition to examining the learning effects of online learning, this meta-analysis has considered
the conditions and practices associated with differences in effectiveness. Conditions are those
features of the context within which the online technology is implemented that are relatively
impervious to change. Conditions include the year in which the intervention took place, the
learners’ demographic characteristics, the teacher’s or instructor’s qualifications, and state
accountability systems. In contrast, practices concern how online learning is implemented (e.g.,
whether or not an online course facilitator is used). In choosing whether or where to use online
learning (e.g., to teach mathematics for high school students, to teach a second language to
elementary students), it is important to understand the degree of effectiveness of online learning
under differing conditions. In deciding how to implement online learning, it is important to
understand the practices that research suggests will increase effectiveness (e.g., community
building among participants, use of an online facilitator, blending work and training).
2

4
• Interactive learning—The learner builds knowledge through inquiry-based collaborative
interaction with other learners; teachers become co-learners and act as facilitators.
This dimension of learning-experience type is closely linked to the concept of learner control
explored by Zhang (2005). Typically, in expository instruction, the technology delivers the
content. In active learning, the technology allows students to control digital artifacts to explore
information or address problems. In interactive learning, technology mediates human interaction
either synchronously or asynchronously; learning emerges through interactions with other
students and the technology.
The learner-control category of interactive learning experiences is related to the so-called “fifth
generation” of distance learning, which stresses a flexible combination of independent and group
learning activities. Researchers are now using terms such as “distributed learning” (Dede 2006)
or “learning communities” to refer to orchestrated mixtures of face-to-face and virtual
interactions among a cohort of learners led by one or more instructors, facilitators or coaches
over an extended period of time (from weeks to years).
Finally, a third characteristic commonly used to categorize online learning activities is the extent
to which the activity is synchronous, with instruction occurring in real time whether in a physical
or a virtual place, or asynchronous, with a time lag between the presentation of instructional
stimuli and student responses. Exhibit 1 illustrates the three dimensions in the framework
guiding this meta-analysis of online learning offerings. The descriptive columns in the table
illustrate uses of online learning comprising dimensions of each possible combination of the
learning experience, synchronicity, and objective (an alternative or a supplement to face-to-face
instruction).

Exhibit 1. Conceptual Framework for Online Learning
Learning
Experience
Dimension Synchronicity
Face-to-Face
Alternative
Face-to-Face
Enhancement
Expository
Synchronous
Live, one-way webcast of online lecture course
with limited learner control (e.g., students
proceed through materials in set sequence)
Viewing webcasts to supplement in-class learning
activities
Asynchronous Math course taught through online video lectures
that students can access on their own schedule
Online lectures on advanced topics made
available as a resource for students in a
conventional math class
Active
Synchronous
Learning how to troubleshoot a new type of
computer system by consulting experts through
live chat
Chatting with experts as the culminating activity for
a curriculum unit on network administration
Asynchronous Social studies course taught entirely through
Web quests that explore issues in U.S. history
Web quest options offered as an enrichment
activity for students completing their regular social
studies assignments early
Interactive
Synchronous
Health-care course taught entirely through an
online, collaborative patient management
simulation that multiple students interact with at
the same time
Supplementing a lecture-based course through a
session spent with a collaborative online
simulation used by small groups of students
Asynchronous
Professional development for science teachers
through “threaded” discussions and message
boards on topics identified by participants
Supplemental, threaded discussions for pre-
service teachers participating in a face-to-face
course on science methods
Exhibit reads: Online learning applications can be characterized in terms of (a) the kind of learning experience they provide, (b) whether
computer-mediated instruction is primarily synchronous or asynchronous and (c) whether they are intended as an alternative or a supplement to
face-to-face instruction.
5

7

conditions and practices were associated with differences in outcomes. For example, they found
that distance education that used synchronous instruction was significantly negative in its effect,
with an average effect size of –0.10, whereas the average effect size for studies using
asynchronous instruction was significantly positive (+0.05). However, the studies that Bernard et
al. categorized as using synchronous communication involved “yoked” classrooms; that is, the
instructor’s classroom was the center of the activity, and one or more distant classrooms
interacted with it in “hub and spoke” fashion. These satellite classes are markedly different from
today’s Web-based communication among the multiple nodes in a “learning network.”
Machtmes and Asher’s earlier (2000) meta-analysis of telecourses sheds light on this issue.10
Although detecting no difference between distance and face-to-face learning overall, they found
results more favorable for telecourses when classrooms had two-way, as opposed to one-way,
interactions.
Although earlier meta-analyses of distance education found it equivalent to classroom instruction
(as noted above), several reviewers have suggested that this pattern may change. They argue that
online learning as practiced in the 21st century can be expected to outperform earlier forms of
distance education in terms of effects on learning (Zhao et al. 2005).
The meta-analysis reported here differs from earlier meta-analyses because its focus has been
restricted to studies that did the following:
• Investigated significant use of the Web for instruction
• Had an objective learning measure as the outcome measure
• Met higher quality criteria in terms of study design (i.e., an experimental or controlled
quasi-experimental design)
Structure of the Report
Chapter 2 describes the methods used in searching for appropriate research articles, in screening
those articles for relevance and study quality, in coding study features, and in calculating effect
sizes. Chapter 3 describes the 51 study effects identified through the article search and screening
and presents findings in the form of effect sizes for studies contrasting purely online or blended
learning conditions with face-to-face instruction. Chapter 4 provides a qualitative narrative
synthesis of research studies comparing variations of online learning interventions. Finally,
chapter 5 discusses the implications of the literature search and meta-analysis for future studies
of online learning and for K–12 online learning practices.

10 Like the present meta-analysis, Machtmes and Asher limited their study corpus to experiments or quasi-
experiments with an achievement measure as the learning outcome.

An effect size is similar to a z-score in that it is expressed in terms of units of standard deviation.
It is defined as the difference between the treatment and control means, divided by the pooled
standard deviation. Effect sizes can be calculated (a) from the means and standard deviations for
the two groups or (b) on the basis of information provided in statistical tests such as t-tests and
analyses of variance. Following the guidelines from the What Works Clearinghouse (2007) and
Lipsey and Wilson (2000), numerical and statistical data contained in the studies were extracted
so that Comprehensive Meta-Analysis software (Biostat Solutions 2006) could be used to
calculate effect sizes (g). The precision of each effect estimate was determined by using the
estimated standard error of the mean to calculate the 95-percent confidence interval for each
effect.
The review of the 99 studies to obtain the data for calculating effect size produced 51
independent effect sizes (28 for Category 1 and 23 for Category 2) from 46 studies. Fifty-three
studies did not report sufficient data to support calculating effect size.
Coding of Study Features
All studies that provided enough data to compute an effect size were coded for their study
features and for study quality. Building on the project’s conceptual framework and the coding
schemes used in several earlier meta-analyses (Bernard et al. 2004; Sitzmann et al. 2006), a
coding structure was developed and pilot-tested with several studies. The top-level coding
structure, incorporating refinements made after pilot testing, is shown in Exhibit A-4 of the
appendix.
To determine interrater reliability, two researchers coded 20 percent of the studies, achieving an
interrater reliability of 86 percent across those studies. Analysis of coder disagreements resulted
in the refinement of some definitions and decision rules for some codes; other codes that
required information that articles did not provide or that proved difficult to code reliably were
eliminated (e.g., whether or not the instructor was certified). A single researcher coded the
remaining studies.
14

15

Data Analysis
Before combining effects from multiple contrasts, effect sizes were weighted to avoid undue
influence of studies with small sample sizes (Hedges and Olkin 1985). For the total set of 51
contrasts and for each subset of contrasts being investigated, a weighted mean effect size
(Hedges’ g+) was computed by weighting the effect size for each study contrast by the inverse of
its variance. The precision of each mean effect estimate was determined by using the estimated
standard error of the mean to calculate the 95 percent confidence interval. Using a fixed-effects
model, the heterogeneity of the effect size distribution (the Q-statistic) was computed to indicate
the extent to which variation in effect sizes was not explained by sampling error alone.
Next, a series of post-hoc subgroup and moderator variable analyses were conducted using the
Comprehensive Meta-Analysis software. A mixed-effects model was used for these analyses to
model within-group variation.13 A between-group heterogeneity statistic (QBetween) was computed
to test for statistical differences in the weighted mean effect sizes for various subsets of the
effects (e.g., studies using blended as opposed to purely online learning for the treatment group).
Chapter 3 describes the results of these analyses.

13 Meta-analysts need to choose between a mixed-effects and a fixed-effects model for investigating moderator
variables. A fixed-effects analysis is more sensitive to differences related to moderator variables, but has a greater
likelihood of producing Type I errors (falsely rejecting the null hypothesis). The mixed-effects model reduces the
likelihood of Type I errors by adding a random constant to the standard errors, but does so at the cost of
increasing the likelihood of Type II errors (incorrectly accepting the null hypothesis). Analysts chose the more
conservative mixed-effects model for this investigation of moderator variables.

3. Findings
This chapter presents the results of the meta-analysis of controlled studies that compared the
effectiveness of online learning with that of face-to-face instruction. The next chapter presents a
narrative synthesis of studies that compared different versions of online learning with each other
rather than with a face-to-face control condition.
Nature of the Studies in the Meta-Analysis
As indicated in chapter 2, 51 independent effect sizes could be abstracted from the study corpus
of 46 studies.14 The number of students in the studies included in the meta-analysis ranged from
16 to 1,857, but most of the studies were modest in scope. Although large-scale applications of
online learning have emerged, only five studies in the meta-analysis corpus included more than
400 learners. The types of learners in these studies were about evenly split between students in
college or earlier years of education and learners in graduate programs or professional training.
The average learner age ranged from 13 to 44. Nearly all the studies involved formal instruction,
with the most common subject matter being medicine or health care. Other content types
included computer science, teacher education, social science, mathematics, languages, science
and business. Roughly half of the learners were taking the instruction for credit or as an
academic requirement. Of the 49 contrasts for which the study indicated the length of instruction,
19 involved instructional time frames of less than a month and the remainder involved longer
periods.
In terms of instructional features, the online learning conditions in these studies were less likely
to be instructor-directed (8 contrasts) than they were to be student-directed, independent learning
(17 contrasts) or interactive and collaborative in nature (23 contrasts). Online learners typically
had opportunities to practice skills or test their knowledge (42 effects were from studies
reporting such opportunities). Opportunities for learners to receive feedback were less common;
however, it was reported in the studies associated with 24 effects. The opportunity for online
learners to have face-to-face contact with the instructor during the time frame of the course was
present in the case of 21 out of 51 effects. The details of instructional media and communication
options available to online learners were absent in many of the study narratives. Among the 51
contrasts, analysts could document the presence of one-way video or audio in the online
condition for 15 effects. Similarly, 16 contrasts involved online conditions with asynchronous
communication only; 9 involved both asynchronous and synchronous online communication; and
26 contrasts came from studies that did not document the types of online communication
provided to learners.

14 After the first literature search, which yielded 29 independent effects, the research team ran additional analyses to
find out how many more studies could be included if the study design criterion were relaxed to include quasi-
experiments with pre- and posttests with no statistical adjustments made for preexisting differences. The relaxed
standard would have increased the corpus for analysis by just 10 studies, nearly all of which were in Category 1
and which had more positive effect sizes than the Category 1 studies with stronger analytic designs. Analysts
decided not to include those studies in the meta-analysis. Instead, the study corpus was enlarged by conducting a
second literature search in July 2008.
17

Among the 51 individual contrasts between online and face-to-face instruction, 11 were
significantly positive, favoring the online or blended learning condition. Two significant
negative effects favored traditional face-to-face instruction. The fact that multiple comparisons
were conducted should be kept in mind when interpreting this pattern of findings. Because
analysts used a α < .05 level of significance for testing differences, one would expect
approximately 1 in 20 contrasts to show a significant difference by chance alone. For 51
contrasts, then, one would expect 2 or 3 significant differences by chance. The finding of 2
significant contrasts favoring face-to-face instruction is clearly within the range one would
expect by chance; the 11 contrasts favoring online or hybrid instruction exceeds what one would
expect by chance.
Exhibit 3 illustrates the 51 effect sizes derived from the 46 articles.15 Exhibits 4a and 4b present
the effect sizes for Category 1 (purely online versus face-to-face) and Category 2 (blended versus
face-to-face) studies, respectively, along with standard errors, statistical significance, and the 95-
percent confidence interval.
Main Effects
The overall finding of the meta-analysis is that classes with online learning (whether taught
completely online or blended) on average produce stronger student learning outcomes than do
classes with solely face-to-face instruction. The mean effect size for all 51 contrasts was +0.24,
p < .001.
The conceptual framework for this study, which distinguishes between purely online and blended
forms of instruction, calls for creating subsets of the effect estimates to address two more
nuanced research questions:
• How does the effectiveness of online learning compare with that of face to-face
instruction? Looking only at the 28 Category 1 effects that compared a purely online
condition with face-to-face instruction, analysts found a mean effect of +0.14, p < .05.
This finding is more positive than those of previous summaries of distance learning
(generally from pre-Internet studies), most of which concluded that learning at a distance
is as effective as classroom instruction but no better.

15 Some references appear twice in Exhibit 3 because multiple effect sizes were extracted from the same article.
Davis et al. (1999) and Caldwell (2006) each included two contrasts—online versus face-to-face (Category 1) and
blended versus face-to-face (Category 2). Rockman et al. (2007) and Schilling et al. (2006) report findings for two
distinct learning measures. Long and Jennings (2005) report findings from two distinct experiments, a “wave 1” in
which teachers were implementing online learning for the first time and a “wave 2” in which teachers
implemented online learning a second time with new groups of students.
18

Exhibit 4a. Purely Online Versus Face-to-Face (Category 1) Studies Included in the Meta-Analysis (continued)

Authors Title Effect Size
95-Percent
Confidence
Interval
Test of Null
Hypothesis
(2-tail)
Retention
Rate
(percentage) Number
of Units
Assigneda

g SE
Lower
Limit
Upper
Limit Z-Value Online
Face-to-
Face
LaRose, Gregg and
Eastin (1998)
Audiographic telecourses for the Web: An
experiment +0.070 0.281 -0.481 0.621 0.25 Unknown Unknown 49 students
Lowry (2007) Effects of online versus face-to-face
professional development with a team-based
learning community approach on teachers’
application of a new instructional practice -0.281 0.335 -0.937 0.370 -0.84 80 93.55 53 students
Mentzer, Cryan and
Teclehaimanot
(2007)
A comparison of face-to-face and Web-based
classrooms
-0.796 0.339 -1.460 -0.131 -2.35* Unknown Unknown 36 students
Nguyen et al.
(2008)
Randomized controlled trial of an Internet-based
versus face-to-face dyspnea self-management
program for patients with chronic obstructive
pulmonary disease: Pilot study +0.292 0.316 -0.327 0.910 0.93 Unknown Unknown
39
participants
Ocker and
Yaverbaum (1999)
Asynchronous computer-mediated
communication versus face-to-face
collaboration: Results on student learning,
quality and satisfaction -0.030 0.214 -0.449 0.389 -0.14 Unknown Unknown 43 students
Padalino and Peres
(2007)
E-learning: A comparative study for knowledge
apprehension among nurses 0.115 0.281 -0.437 0.666 0.41 Unknown Unknown
49
participants
Peterson and Bond
(2004)
Online compared to face-to-face teacher
preparation for learning standards-based
planning skills +0.100 0.214 -0.320 0.520 0.47 Unknown Unknown 4 sections
Schmeeckle (2003) Online training: An evaluation of the
effectiveness and efficiency of training law
enforcement personnel over the Internet
-0.106 0.198 -0.494 0.282 -0.53 Unknown Unknown 101 students
Schoenfeld-Tacher,
McConnell and
Graham (2001)
Do no harm: A comparison of the effects of
online vs. traditional delivery media on a science
course +0.800 0.459 -0.100 1.700 1.74 100 99.94 Unknown
22

Exhibit 4a: Purely Online versus Face-to-Face (Category 1) Studies Included in the Meta-analysis (continued)

Authors Title Effect Size
95-Percent
Confidence
Interval
Test of Null
Hypothesis
(2-tail)
Retention
Rate
(percentage) Number
of Units
Assigneda

g SE
Lower
Limit
Upper
Limit Z-Value Online
Face-to-
Face
Sexton, Raven and
Newman (2002)
A comparison of traditional and World Wide
Web methodologies, computer anxiety, and
higher order thinking skills in the inservice
training of Mississippi 4-H extension agents -0.422 0.385 -1.177 0.332 -1.10 Unknown Unknown 26 students
Sun, Lin and Yu
(2008)
A study on learning effect among different
learning styles in a Web-based lab of science for
elementary school students +0.180 0.187 -0.187 0.547 0.96 Unknown Unknown 4 classrooms
Turner et al. (2006) Web-based learning versus standardized
patients for teaching clinical diagnosis: A
randomized, controlled, crossover trial +0.242 0.367 -0.477 0.960 0.66 Unknown Unknown 30 students
Vandeweerd et al.
(2007)
Teaching veterinary radiography by e-learning
versus structured tutorial: A randomized, single-
blinded controlled trial +0.144 0.207 -0.262 0.550 0.70 Unknown Unknown 92 students
Wallace and
Clariana (2000)
Achievement predictors for a computer-
applications module delivered online +0.109 0.206 -0.295 0.513 0.53 Unknown Unknown 4 sections
Wang (2008) Developing and evaluating an interactive
multimedia instructional tool: Learning outcomes
and user experiences of optometry students -0.071 0.136 -0.338 0.195 -0.53 Unknown Unknown 4 sectionsc
Zhang (2005) Interactive multimedia-based e-learning: A study
of effectiveness +0.381 0.339 -0.283 1.045 1.12 Unknown Unknown 51 students
Zhang et al. (2006) Instructional video in e-learning: Assessing the
effect of interactive video on learning
effectiveness +0.499 0.244 0.022 0.977 2.05* Unknown Unknown 69 students
Exhibit reads: The effect size for the Hugenholtz et al. (2008) study of online medical education was +0.11, which was not significantly different from 0.
*p < .05, ** p < .01, SE = Standard error
a The number given represents the assigned units at study conclusion. It excludes units that attrited.
b Two outcome measures were used to compute one effect size. The first outcome measure was completed by 17 participants, and the second outcome measure was
completed by 20 participants.
c This study is a crossover study. The number of units represents those assigned to treatment and control conditions in the first round.
23

Exhibit 4b. Blended Versus Face-to-Face (Category 2) Studies Included in the Meta-Analysis
Authors Title Effect Size
95-Percent
Confidence
Interval
Test of Null
Hypothesis
(2-tail)
Retention
Rate
(percentage)
Number
of Units
Assigneda g SE
Lower
Limit
Upper
Limit Z-Value Online
Face-
to-
Face
Aberson et al.
(2003)
Evaluation of an interactive tutorial for teaching
hypothesis testing concepts +0.700 0.404 -0.092 1.492 1.73 Unknown
.75
2 sections
Al-Jarf (2004) The effects of Web-based learning on struggling
EFL college writers +0.740 0.194 0.360 1.120 3.82*** Unknown Unknown 113 students
Caldwell (2006) A comparative study of traditional, Web-based
and online instructional modalities in a computer
programming course +0.251 0.311 -0.359 0.861 0.81 100 100 60 students
Davis et al. (1999) Developing online courses: A comparison of
Web-based instruction with traditional instruction +0.335 0.338 -0.327 0.997 0.99 Unknown Unknown
2 courses/
classrooms
Day, Raven and
Newman (1998)
The effects of World Wide Web instruction and
traditional instruction and learning styles on
achievement and changes in student attitudes in
a technical writing in agricommunication course +1.113 0.289 0.546 1.679 3.85*** 89.66 96.55 2 sections
DeBord, Aruguete
and Muhlig (2004)
Are computer-assisted teaching methods
effective?
+0.130 0.188 -0.239 0.499 0.69
Unknown Unknown
112 students
El-Deghaidy and
Nouby (2008)
Effectiveness of a blended e-learning
cooperative approach in an Egyptian teacher
education program +0.475 0.386 -0.282 1.232 1.23 Unknown Unknown 26 students
Englert et al. (2007) Scaffolding the writing of students with
disabilities through procedural facilitation using
an Internet-based technology
+0.740 0.345 0.064 1.416 2.15* Unknown Unknown
6 classrooms
from
5 urban
schools
Frederickson, Reed
and Clifford (2005)
Evaluating Web-supported learning versus
lecture-based teaching: Quantitative and
qualitative perspectives +0.138 0.345 -0.539 0.814 0.40 Unknown Unknown 2 sections
Gilliver, Randall and
Pok (1998) Learning in cyberspace: Shaping the future +0.477 0.111 0.260 0.693 4.31*** Unknown Unknown 24 classes
Long and Jennings
(2005) [Wave 1] c
The effect of technology and professional
development on student achievement +0.025 0.046 -0.066 0.116 0.53 Unknown Unknown 9 schools
24

Practice Variables
Exhibit 5 shows the variation in effectiveness associated with 12 practice variables. Analysis of
these variables addresses the third research question:
What practices are associated with more effective online learning?
Exhibit 5 and the two data exhibits that follow show significance results both for the various
subsets of studies considered individually and for the test of the dimension used to subdivide the
study sample (i.e., the potential moderator variable). For example, in the case of Computer-
Mediated Communication With Peers, both the 17 contrasts in which students in the online
condition had only asynchronous communication with peers and the 7 contrasts in which online
students had both synchronous and asynchronous communication with peers are shown in the
table. The two subsets had mean effect sizes of +0.27 and +0.32, respectively, and both were
statistically different from 0. The Q-statistic of homogeneity tests whether the variability in
effect sizes for these contrasts is associated with the type of peer communication available. The
Q-statistic for Computer-Mediated Communication With Peers (0.13) is not statistically different
from 0, which is unsurprising because both studies of online learning with only asynchronous
communication and those with both asynchronous and synchronous communication found
similar positive effects on average.
The test of the moderator variable most central to this study—whether a blended online condition
including face-to-face elements is associated with greater advantages over classroom instruction
than is pure online learning—was discussed above. As noted there, the effect size for blended
approaches contrasted against face-to-face instruction is larger than that for purely online
approaches contrasted against face-to-face instruction. The other two practice variables included
in the chapter 1 conceptual framework—learning experience type and synchronous versus
asynchronous communication with the instructor—were tested in a similar fashion. Neither was
found to moderate significantly the size of the online learning effect. However, examination of
the learning experience study subsets indicated that the mean effect size for studies where the
online learning was instructor-directed expository (+0.36) and the mean effect size for
collaborative, interactive instruction (+0.28) were significantly positive whereas the mean effect
size for independent, active online learning (+0.15) was not.16
Among the other 10 practices, which were not part of the conceptual model, only the amount of
time that students in the treatment condition spent on task compared with students in the face-to-
face condition proved to be a significant moderator.17 The mean effect size for studies with more
time spent on task by online learners than learners in the control condition was +0.46 compared
with +0.19 for studies in which the learners in the face-to-face condition spent as much time or
more on task (Q = 3.88, p < .05).

16 Online experiences in which students explored digital artifacts and controlled the specific material they wanted to
view were categorized as “active” learning experiences.
17 If the five K-12 studies are dropped from the meta-analysis corpus, the p value for this moderator variable rises to
p < .06.
28

Exhibit 5. Tests of Practices as Moderator Variables
Variable Contrast
Number
Studies
Weighted
Effect Size
Standard
Error
Lower
Limit
Upper
Limit Q-Statistic
Pedagogy/learning
experiencea
Instructor-directed
(expository) 8 0.363** 0.115 0.138 0.588
3.03 Independent (active) 17 0.145 0.077 -0.005 0.296
Collaborative
(interactive) 23 0.283*** 0.070 0.146 0.419
Computer-
mediated
communication
with instructor a
Asynchronous only 16 0.305*** 0.095 0.120 0.491
0.97 Synchronous +
Asynchronous 9 0.153 0.123 -0.089 0.394
Computer-
mediated
communication
with peersa
Asynchronous only 17 0.268*** 0.079 0.113 0.422
0.13
Synchronous +
Asynchronous 7 0.321** 0.125 0.076 0.567
Treatment
durationa
Less than 1 month 19 0.227** 0.082 0.066 0.389
0.07
More than 1 month 30 0.255*** 0.063 0.132 0.378
Media featuresa
Text-based only 15 0.281** 0.100 0.086 0.477
0.13
Text + other media 32 0.239*** 0.060 0.121 0.357
Time on taska
Online > Face to
Face 10 0.461*** 0.110 0.246 0.676
3.88*
Same or Face to
Face > Online 17 0.189* 0.084 0.025 0.353
One-way video or
audio
Present 15 0.118 0.082 -0.043 0.279
3.62
Absent/Not reported 36 0.308*** 0.057 0.196 0.421
Computer-based
instruction
elements
Present 30 0.263*** 0.061 0.144 0.382
0.20
Absent/Not reported 21 0.220** 0.077 0.069 0.371
Opportunity for
face-to-face time
with instructor
During instruction 21 0.277*** 0.069 0.142 0.411
0.37 Before or after
instruction 12 0.220* 0.108 0.009 0.431
Absent/Not reported 18 0.217* 0.086 0.047 0.386
Opportunity for
face-to-face time
with peers
During instruction 21 0.274*** 0.068 0.141 0.408
0.94 Before or after
instruction 13 0.160 0.102 -0.040 0.359
Absent/Not reported 17 0.266** 0.089 0.091 0.442
Opportunity to
practice
Present 42 0.264*** 0.052 0.161 0.366
0.65
Absent/Not reported 9 0.159 0.118 -0.072 0.391
Feedback
provided
Present 24 0.248*** 0.072 0.107 0.388
0.00
Absent/Not reported 27 0.247*** 0.065 0.118 0.375
Exhibit reads: Studies in which time spent in online learning exceeded time in the face-to-face condition had a mean
effect size of +0.46 compared with +0.19 for studies in which face-to-face learners had as much or more instructional
time.
*p < .05, **p < .01, ***p < .001.
a The moderator analysis for this variable excluded studies that did not report information for this feature.
29

Condition Variables
The strategy to investigate whether study effect sizes varied with publication year, which was
taken as a proxy for the sophistication of available technology, involved splitting the study
sample into two nearly equal subsets by contrasting studies published between 1997 and 2003
against those published in 2004 through July 2008.
The studies were divided into three subsets of learner type: K–12 students, undergraduate and
community college students (the largest single group), and other types of learners (graduate
students or individuals receiving job-related training). As noted above, the studies covered a
wide range of subjects, but medicine and health care were the most common. Accordingly, these
studies were contrasted against studies in other fields. Tests of these conditions as potential
moderator variables addressed the study’s fourth research question:
What conditions influence the effectiveness of online learning?
None of the three conditions tested emerged as a statistically significant moderator variable. In
other words, for the range of student types for which studies are available, online learning
appeared more effective than traditional face-to-face instruction in both older and newer studies,
with undergraduate and older learners, and in both medical and other subject areas. Exhibit 6
provides the results of the analysis of conditions.
Exhibit 6. Tests of Conditions as Moderator Variables
Variable Contrast
Number of
Contrasts
Weighted
Effect
Size
Standard
Error
Lower
Limit
Upper
Limit Q-Statistic
Year
Published
1997–2003 14 0.266** 0.095 0.080 0.453
0.06
2004 or after 37 0.240*** 0.055 0.133 0.347
Learner
Type
K–12 students 7 0.158 0.101 -0.040 0.356
3.70 Undergraduate 25 0.345*** 0.069 0.209 0.480
Graduate
student/Other 19 0.172* 0.077 0.021 0.324
Subject
Matter
Medical/ Health
care 16 0.302*** 0.084 0.136 0.467 0.63
Other 35 0.221*** 0.057 0.110 0.332
Exhibit reads: The positive effect associated with online learning over face-to-face instruction was
significant both for studies published between 1997 and 2003 and for those published in 2004 or later; the
effect size does not vary significantly with period of publication.
*p < .05, **p < .01, ***p < .001.
30

Exhibit 7. Studies of Online Learning Involving K-12 Students (continued)
Long and Jennings (2005) examined whether the performance of eighth-grade students whose
teachers integrated the use of the Pathways to Freedom Electronic Field Trips—an online
collection of interactive activities designed by Maryland Public Television—improved compared
with performance of students whose teachers taught the same content without the online
materials. The study provided two sets of analyses from two waves of data collection, yielding
two independent effect sizes. The first set of analyses involved the data from nine schools in two
Maryland districts. Schools were assigned randomly to conditions. Teachers in both conditions
covered the same learning objectives related to slavery and the Underground Railroad, with the
treatment teachers using the Pathways to Freedom Electronic Field Trips materials. A small
effect size of +0.03 favoring the online condition was computed from change scores on
researcher-developed multiple-choice tests administered to 971 students.
Long and Jennings’ (2005, wave 2) second study involved a subset of teachers from one of the
two participating districts, which was on a semester schedule. The teachers from this district
covered the same curriculum twice during the year for two different sets of students. The gain
scores of 846 students of six teachers (three treatment teachers and three control teachers) from
both semesters were collected. Regression analysis indicated an effect size of +0.55 favoring the
online conditions. This study also looked into the maturation effects of teachers’ using the online
materials for the second time. As hypothesized, the results showed that the online materials were
used more effectively in the second semester.
Sun, Lin and Yu (2008) conducted a quasi-experimental study to examine the effectiveness of a
virtual Web-based science lab with 113 fifth-grade students in Taiwan. Although both treatment
and control groups received an equal number of class hours and although both groups conducted
manual experiments, students in the treatment condition used the virtual Web-based science lab
for part of their lab time. The Web-based lab enabled students to conduct virtual experiments
while teachers observed student work and corrected errors online. The control group students
conducted equivalent experiments using conventional lab equipment. Matched pre- and posttest
scores on researcher-developed assessments were collected for a total of 113 students (56 from
the treatment group and 57 from the comparison group) in four classrooms from two randomly
sampled schools. An effect size of +0.18 favoring the virtual lab condition was obtained from
analysis of covariance results, controlling for pretest scores.
A small-scale quasi-experiment was conducted by Englert et al. (2007). This study examined the
effectiveness of a Web-based writing support program with 35 elementary-age students from six
special education classrooms across five urban schools. Students in the treatment group used a
Web-based program that supported writing performance by prompting attention to the topical
organization and structure of ideas during the planning and composing phases of writing. Control
students used similar writing tools provided in traditional paper-and-pencil formats. Pre- and
posttests of student writing, scored on a researcher-developed rubric, were used as outcome
measures. An effect size of +0.74 favoring the online condition was obtained from an analysis of
covariance controlling for writing pretest scores.
33

Exhibit 8. Tests of Study Features as Moderator Variables
Variable Contrast
Number
of
Studies
Weighted
Effect
Size
Standard
Error
Lower
Limit
Upper
Limit Q-Statistic
Sample size
Fewer than 35 11 0.312* 0.133 0.051 0.574
0.28 From 35 to 100 21 0.240** 0.080 0.083 0.396
More than 100 19 0.235*** 0.066 0.106 0.364
Type of
knowledge
testeda
Declarative 13 0.191* 0.089 0.017 0.365
1.08
Procedural/
Procedural and
declarative
29 0.293*** 0.065 0.166 0.419
Strategic
knowledge 5 0.335* 0.158 0.025 0.644
Study design
Random
assignment
control
33 0.279*** 0.061 0.158 0.399
0.71
Quasi-
experimental
design with
statistical control
13 0.203* 0.091 0.025 0.381
Crossover
design 5 0.178 0.151 -0.117 0.473
Unit of
assignment
to conditionsa
Individual 33 0.207*** 0.060 0.088 0. 325
5.18 Class section 7 0.517*** 0.129 0.264 0.770
Course/School 9 0.190* 0.093 0.009 0.372
Instructor
equivalencea
Same instructor 20 0.227** 0.072 0.086 0.368
0.67 Different
instructor 20 0.146* 0.068 0.013 0.280
Equivalence
of curriculum/
instructiona
Identical/
Almost identical 30 0.200*** 0.056 0.091 0.309
5.40* Different/
Somewhat
different
17 0.418*** 0.075 0.270 0.566
Exhibit reads: The average effect size was significantly positive for studies with a sample size of less
than 35 as well as for those with 35 to 100 and those with a sample size larger than 100; the weighted
average effect did not vary with size of the study sample.
*p < .05, **p < .01, ***p < .001.
a The moderator analysis excluded some studies because they did not report information about this
feature.
34

35

Effect sizes do not vary depending on whether or not the same instructor or instructors taught in
the face-to-face and online conditions (Q = 0.67, p > .05). The average effect size for the 20
contrasts in which instructors were the same across conditions was +0.23, p < .01. The average
effect size for contrasts in which instructors varied across conditions was +0.15, p < .05. The
only study method variable that proved to be a significant moderator of effect size was
comparability of the instructional materials and approach for treatment and control students.
The analysts coding study features examined the descriptions of the instructional materials and
the instructional approach for each study and coded them as “identical,” “almost identical,”
“different” or “somewhat different” across conditions. Adjacent coding categories were
combined (creating the two study subsets Identical/Almost Identical and Different/Somewhat
Different) to test Equivalence of Curriculum/Instruction as a moderator variable. Equivalence of
Curriculum/Instruction was a significant moderator variable (Q = 5.40, p < .05). An examination
of the study subgroups shows that the average effect for studies in which online learning and
face-to-face instruction were described as identical or nearly so was +0.20, p < .01, compared
with an average effect of +0.42 (p < .001) for studies in which curriculum materials and
instructional approach varied across conditions.
A marginally significant effect was found for the unit assigned to treatment and control
conditions. Effects tended to be smaller in studies in which individual students or sections, rather
than whole courses or schools, were assigned to online and face-to-face conditions (Q = 5.18,
p < .10)18.
The moderator variable analysis for aspects of study method also found additional patterns that
did not attain statistical significance but that should be re-tested once the set of available rigorous
studies of online learning has expanded. The type of learning outcome tested, for example, may
influence the magnitude of effect sizes. Thirteen studies measured declarative knowledge
outcomes only, typically through multiple-choice tests. A larger group of studies (29) looked at
students’ ability to perform a procedure, or they combined procedural and declarative knowledge
outcomes in their learning measure. Five studies used an outcome measure that focused on
strategic knowledge. (Four studies did not describe their outcome measures in enough detail to
support categorization.) Among the subsets of studies, the average effect for studies that included
procedural knowledge in their learning outcome measure (effect size of +0.29) and that for
studies that measured strategic knowledge (effect size of +0.34) appeared larger than the mean
effect size for studies that used a measure of declarative knowledge only (+0.19). Even so, the
Type of Knowledge Tested was not a significant moderator variable (Q = 1.08, p > .05).

18 This moderator variable is statistically significant if the five K-12 studies are excluded from the analysis.

4. Narrative Synthesis of Studies
Comparing Variants of Online Learning
This chapter presents a narrative summary of Category 3 studies—those that examined the
learning effects of variations in online practices such as different versions of blended instruction
or online learning with and without immediate feedback to the learner. The literature search and
screening (described in chapter 2) identified 84 Category 3 studies reported in 79 articles.19
Within the set of Category 3 studies, five used K–12 students as subjects and 10 involved K–12
teacher education or professional development. College undergraduates constituted the most
common learner type (see Exhibit 9). All Category 3 studies involved formal education. Course
content for Category 3 studies covered a broad range of subjects, including observation skills,
understanding Internet search engines, HIV/AIDS knowledge and statistics.
When possible, the treatment manipulations in Category 3 studies were coded using the practice
variable categories that were used in the meta-analysis to facilitate comparisons of findings
between the meta-analysis and the narrative synthesis. No attempt was made to statistically
combine Category 3 study results, however, because of the wide range of conditions compared in
the different studies.

Exhibit 9. Learner Types for Category 3 Studies
Educational Level Number of Studies
K–12 5
Undergraduate 37
Graduate 4
Medicala 18
Teacher professional developmentb 10
Adult training 4
Otherc 4
Not available 2
Total 84
Exhibit reads: K–12 students were the learners in 5 of the 84 studies of
alternative online practices.
a The medical category spans undergraduate and graduate educational levels
and includes nursing and related training.
b Teacher professional development includes preservice and inservice training.
c The Other category includes populations consisting of a combination of
learner types such as student and adult learners or undergraduate and
graduate learners.

19 Some articles contained not only contrasts that fit the criteria for Category 1 or 2 but also contrasts that fit
Category 3. The appropriate contrasts between online and face-to-face conditions were used in the meta-analysis;
the other contrasts were reviewed as part of the Category 3 narrative synthesis presented here.
37

of online and face-to-face versus online only). For example, the lecturer in the Keefe (2003)
study may have covered material not available to the students reviewing the lecture’s PowerPoint
slides online. Alternately, in the Poirier and Feldman (2004) study, students interacting with the
instructor in two online discussions a week may have received more content than did those
receiving face-to-face lectures.
Davis et al. (1999) attempted to equate the content delivered in their three class sections (online,
traditional face-to-face, and a blended condition in which students and instructor met face-to-
face but used the online modules). Students in an educational technology course were randomly
assigned to one of the three sections. No significant differences among the three conditions were
found in posttest scores on a multiple-choice test.
An additional six studies contrasting purely online conditions and blended conditions (without
necessarily equating learning content across conditions) also failed to find significant differences
in student learning. Ruchti and Odell (2002) compared test scores from two groups of students
taking a course on elementary science teaching methods. One group took online modules; the
other group received instruction in a regular class, supplemented with an online discussion board
and journal (also used in the online course condition). No significant difference between the
groups was found.
Beile and Boote (2002) compared three groups: one with face-to-face instruction alone, another
with face-to-face instruction and a Web-based tutorial, and a third with Web-based instruction
and the same Web-based tutorial. The final quiz on library skills indicated no significant
differences among conditions.
Gaddis et al. (2000) compared composition students’ audience awareness between a blended
course and a course taught entirely online. The same instructor taught both groups, which also
had the same writing assignments. Both groups used networked computers in instruction, in
writing and for communication. However, the “on campus” group met face-to-face, giving
students the opportunity to communicate in person, whereas the “off campus” group met only
online. The study found no significant difference in learner outcomes between the two groups.
Similarly, Caldwell (2006) found no significant differences in performance on a multiple-choice
test between undergraduate computer science majors enrolled in a blended course and those
enrolled in an online course. Both groups used a Web-based platform for instruction, which was
supplemented by a face-to-face lab component for the blended group.
Scoville and Buskirk (2007) examined whether the use of traditional or virtual microscopy
would affect learning outcomes in a medical histology course. Students were assigned to one of
four sections: (a) a control section where learning and testing took place face-to-face, (b) a
blended condition where learning took place virtually and the practical examination took place
face-to-face, (c) a second blended condition where learning took place face-to-face and testing
took place virtually, and (d) a fully online condition. Scoville and Buskirk found no significant
differences in unit test scores by learning groups.
Finally, McNamara et al. (2008) studied the effectiveness of different approaches to teaching a
weight-training course. They divided students into three groups: a control group that received
39

In summary, many researchers have hypothesized that the addition of images, graphics, audio,
video or some combination would enhance student learning and positively affect achievement.
However, the majority of studies to date have found that these media features do not affect
learning outcomes significantly.
Learning Experience Type
Other Category 3 studies manipulated different features of the online learning environment to
investigate the effects of learner control or type of learning experience. The learning experience
studies provide some evidence that suggests an advantage for giving learners an element of
control over the online resources with which they engage; however, the studies’ findings are
mixed with respect to the relative effectiveness of the three learning experience types in the
conceptual framework presented in chapter 2.
Four studies (Cavus et al. 2007; Dinov, Sanchez and Christou 2008; Gao and Lehman 2003;
Zhang 2005) provide preliminary evidence supporting the hypothesis that conditions in which
learners have more control of their learning (either active or interactive learning experiences in
our conceptual framework) produce larger learning gains than do instructor-directed conditions
(expository learning experiences). Three other studies failed to find such an effect (Cook et al.
2007; Evans 2007; Smith 2006).
Zhang (2005) reports on two studies comparing expository learning with active learning, both of
which found statistically positive results in favor of active learning. Zhang manipulated the
functionality of a Web course to create two conditions. For the control group, video and other
instruction received over the Web had to be viewed in a specified order, videos had to be viewed
in their entirety (e.g., a student could not fast forward) and rewinding was not allowed. The
treatment group could randomly access materials, watching videos in any sequence, rewinding
them and fast forwarding through their content. Zhang found a statistically significant positive
effect in favor of learner control over Web functionality (see also the Zhang et al. 2006 study
described above). Gao and Lehman (2003) found that students who were required to complete a
“generative activity” in addition to viewing a static Web page performed better on a test about
copyright law than did students who viewed only the static Web page. Cavus, Uzonboylu and
Ibrahim (2007) compared the success rates of students learning the Java programming language
who used a standard collaborative tool with the success rate of those who used an advanced
collaborative tool that allowed compiling, saving and running programs inside the tool. The
course grades for students using the advanced collaborative tool were higher than those of
students using the more standard tool. Similarly, Dinov, Sanchez and Christou (2008) integrated
tools from the Statistics Online Computational Resource in three courses in probability and
statistics. For each course, two groups were compared: one group of students received a “low-
intensity” experience that provided them with access to a few online statistical tools; the other
students received a “high-intensity” condition with access to many online tools for acting on
data. Across the three classes, pooling all sections, students in the more active, high-intensity
online tool condition demonstrated better understanding of the material on mid-term and final
examinations than did the other students.
These studies that found positive effects for learner control and nondidactic forms of instruction
are counterbalanced by studies that found mixed or null effects from efforts to provide a more
41

expository material) with a condition that included no simulation. Castaneda also manipulated
the sequencing of instructional activities, with the interaction with the simulation coming either
before or after completion of the expository portion of the instructional module. Knowledge
gains from pre- to posttest were greater for students with either type of simulation, provided they
were exposed to it after, rather than before, the expository instruction.
Hibelink (2007) explored the effectiveness of using two-dimensional versus three-dimensional
images of human anatomy in an online undergraduate human anatomy lab. The group of students
that used three-dimensional images had a small, but significant advantage in identifying
anatomical parts and spatial relationships. Contrasting results were obtained by Loar (2007) in an
examination of the effects of computer-based case study simulations on students’ diagnostic
reasoning skills in nurse practitioner programs. All groups received identical online lectures,
followed by an online text-based case study for one group and by completion of a computer-
simulated case study for the other. No difference was found between the group receiving the case
simulation versus that receiving the text-based version of the same case.
Individualized Instruction
The online learning literature has also explored the effects of using computer-based instruction
elements to individualize instruction so that the online learning module or platform responds
dynamically to the participant’s questions, needs or performance. There were only two online
learning studies of the effects of individualizing instruction, but both found a positive effect.
Nguyen (2007) compared the experiences of people learning to complete tax preparation
procedures, contrasting those who used more basic online training with those who used an
enhanced interface that incorporated a context-sensitive set of features, including integrated
tutorials, expert systems, and content delivered in visual, aural and textual forms. Nguyen found
that this combination of enhancements had a positive effect.
Grant and Courtoreille (2007) studied the use of post-unit quizzes presented either as (a) fixed
items that provided feedback only about whether or not the student’s response was correct or (b)
post-unit quizzes that gave the student the opportunity for additional practice on item types that
had been answered incorrectly. The response-sensitive version of the tutorial was found to be
more effective than the fixed-item version, resulting in greater changes between pre- and posttest
scores.
Supports for Learner Reflection
Nine studies (Bixler 2008; Chang 2007; Chung, Chung and Severance 1999; Cook et al. 2005;
Crippen and Earl 2007; Nelson 2007; Saito and Miwa 2007; Shen, Lee and Tsai 2007; Wang et
al. 2006) examined the degree to which promoting aspects of learner reflection in a Web-based
environment improved learning outcomes. These studies found that a tool or feature prompting
students to reflect on their learning was effective in improving outcomes.
For example, Chung, Chung and Severance (1999) examined how computer prompts designed to
encourage students to use self-explanation and self-monitoring strategies affected learning, as
measured by students’ ability to integrate ideas from a lecture into writing assignments. Chung et
44

al. found that students in the group receiving the computer prompts integrated and elaborated a
significantly higher number of the concepts in their writing than did those in the control group.
In a quasi-experimental study of Taiwan middle school students taking a Web-based biology
course, Wang et al. (2006) found that students in the condition using a formative online self-
assessment strategy performed better than those in conditions using traditional tests, whether the
traditional tests were online or administered in paper-and-pencil format. In the formative online
assessment condition, when students answered an item incorrectly, they were told that their
response was not correct, and they were given additional resources to explore to find the correct
answer. (They were not given the right answer.) This finding is similar to that of Grant and
Courtoreille (2007) described above.
Cook et al. (2005) investigated whether the inclusion of “self-assessment” questions at the end of
modules improved student learning. The study used a randomized, controlled, crossover trial, in
which each student took four modules, two with the self-assessment questions and two without.
The order of modules was randomly assigned. Student performance was statistically higher on
tests taken immediately after completion of modules that included self-assessment questions than
after completion of those without such questions—an effect that the authors attributed to the
stimulation of reflection. This effect, however, did not persist on an end-of-course test, on which
all students performed similarly.
Shen, Lee and Tsai (2007) found a combination of effects for self-regulation and opportunities to
learn through realistic problems. They compared the performance of students who did and did
not receive instruction in self-regulation learning strategies such as managing study time, goal-
setting and self-evaluation. The group that received instruction in self-regulated learning
performed better in their online learning.
Bixler (2008) examined the effects of question prompts asking students to reflect on their
problem-solving activities. Crippen and Earl (2007) investigated the effects of providing students
with examples of chemistry problem solutions and prompts for students to provide explanations
regarding their work. Chang (2007) added a self-monitoring form for students to record their
study time and environment, note their learning process, predict their test scores and create a
self-evaluation. Saito and Miwa (2007) investigated the effects of student reflection exercises
during and after online learning activities. Nelson (2007) added a learning guidance system
designed to support a student’s hypothesis generation and testing processes without offering
direct answers or making judgments about the student’s actions. In all of these studies, the
additional reflective elements improved students’ online learning.
Overall, the available research evidence suggests that promoting self-reflection, self-regulation
and self-monitoring leads to more positive online learning outcomes. Features such as prompts
for reflection, self-explanation and self-monitoring strategies have shown promise for improving
online learning outcomes.
45

Moderating Online Groups
Organizations providing or promoting online learning generally recommend the use of
instructors or other adults as online moderators, but research support for the effects of this
practice on student learning is mixed. A study by Bernard and Lundgren-Cayrol (2001) suggests
that instructor moderation may not improve learning outcomes in all contexts. The study was
conducted in a teacher education course on educational technology in which the primary
pedagogical approach was collaborative, project-based learning. Students in the course were
randomly assigned to groups receiving either low or high intervention on the part of a moderator
and composed of either random or self-selected partners. The study did not find a main effect for
moderator intervention. In fact, the mean examination scores of the low-moderation, random-
selection groups were significantly higher than those of the other groups. A study by De Wever,
Van Winckel and Valcke (2008) also found mixed effects resulting from instructor moderation.
This study was conducted during a clinical rotation in pediatrics in which knowledge of patient
management was developed through case-based asynchronous discussion groups. Researchers
used a crossover design to create four conditions based on two variables: the type of moderator
(instructor moderator versus student moderator) and the presence of a developer of alternatives
for patient management (assigned developer versus no assigned developer). The presence of a
course instructor as moderator was found not to improve learning outcomes significantly. When
no assigned developer of alternatives was assigned, the two moderator conditions performed
equivalently. When a developer of alternatives was specified, the student-moderated groups
performed significantly better than the instructor-moderated groups.
Alternately, Zhang (2004) found that an externally moderated group scored significantly higher
on problems calling for use of statistical knowledge and problem-solving skills than a peer-
controlled group on both well- and ill-structured problems. Zhang’s study compared the
effectiveness of peer versus instructor moderation of online asynchronous collaboration.
Students were randomly assigned to one of two groups. One group had a “private” online space
where students entirely controlled discussion. The other group’s discussion was moderated by
the instructor, who also engaged with students through personal e-mails and other media.
Scripts for Online Interaction
Four Category 3 studies investigated alternatives to human moderation of online discussion in
the form of “scaffolding” or “scripts” designed to produce more productive online interaction.
The majority of these studies indicated that the presence of scripts to guide interactions among
groups learning together online did not appear to improve learning outcomes.
The one study that found positive student outcomes for learners who had been provided scripts
was conducted by Weinberger et al. (2005). These researchers created two types of scripts:
“epistemic scripts,” which specified how learners were to approach an assigned task and guided
learners to particular concepts or aspects of an activity, and “social scripts,” which structured
how students should interact with each other through methods such as gathering information
from each other by asking critical questions. They found that social scripts improved
performance on tests of individual knowledge compared with a control group that participated in
online discussions without either script (whether or not the epistemic script was provided).
46

49
Finally, readers should be cautioned that the literature on alternative online learning practices has
been conducted for the most part by professors and other instructors who are conducting research
using their own courses. Moreover, the combinations of technology, content and activities used
in different experimental conditions have often been ad hoc rather than theory based. As a result,
the field lacks a coherent body of linked studies that systematically test theory-based approaches
in different contexts.

5. Discussion and Implications
The meta-analysis reported here differs from prior meta-analyses of distance learning in several
important respects:
• Only studies of Web-supported learning have been included.
• All effects have been based on objective measures of learning.
• Only studies with controlled designs that met minimum quality criteria have been
included.
The corpus of 51 effect sizes extracted from 46 studies meeting these criteria was sufficient to
demonstrate that in recent applications, online learning has been modestly more effective, on
average, than the traditional face-to-face instruction with which it has been compared.
The test for homogeneity of effects found significant variability in the effect sizes for the
different online learning studies, justifying a search for moderator variables that could explain
the differences in outcomes. The moderator variable analysis found only three moderators
significant at p < .05. Effects were larger when a blended rather than a purely online condition
was compared with face-to-face instruction; when students in the online condition spent more
time learning than did students in the face-to-face condition; and when the curricular materials
and instruction varied between the online and face-to-face conditions. This pattern of significant
moderator variables is consistent with the interpretation that the advantage of online conditions
in these recent studies stems from aspects of the treatment conditions other than the use of the
Internet for delivery per se.
Clark (1983) has cautioned against interpreting studies of instruction in different media as
demonstrating an effect for a given medium inasmuch as conditions may vary with respect to a
whole set of instructor and content variables. That caution applies well to the findings of this
meta-analysis, which should not be construed as demonstrating that online learning is superior as
a medium. Rather, it is the combination of elements in the treatment conditions, which are likely
to include additional learning time and materials as well as additional opportunities for
collaboration, that has proven effective. The meta-analysis findings do not support simply
putting an existing course online, but they do support redesigning instruction to incorporate
additional learning opportunities online.
Several practices and conditions associated with differential effectiveness in distance education
meta-analyses (most of which included nonlearning outcomes such as satisfaction) were not
found to be significant moderators of effects in this meta-analysis of Web-based online learning.
Nor did tests for the incorporation of instructional elements of computer-based instruction (e.g.,
online practice opportunities and feedback to learners) find that these variables made a
difference. Online learning conditions produced better outcomes than face-to-face learning alone,
regardless of whether these instructional practices were used.
The meta-analysis did not find differences in average effect size between studies published
before 2004 (which might have used less sophisticated Web-based technologies than those
51

available since) and studies published from 2004 on (possibly reflecting the more sophisticated
graphics and animations or more complex instructional designs available). Nor were differences
associated with the nature of the subject matter involved.
Finally, the examination of the influence of study method variables found that effect sizes did not
vary significantly with study sample size or with type of design. It is reassuring to note that, on
average, online learning produced better student learning outcomes than face-to-face instruction
in those studies with random-assignment experimental designs (p < .001) and in those studies
with the largest sample sizes (p < .001).
The relatively small number of studies meeting criteria for inclusion in this meta-analysis limits
the power of tests for moderator variables. A few contrasts that did not attain significance (e.g.,
learning experience or type of knowledge tested) might have emerged as significant influences
under a fixed-effects analysis and may prove significant when tested in future meta-analyses
with a larger corpus of studies.
The narrative synthesis of studies comparing variations of online learning provides some
additional insights with respect to designing effective online learning experiences. The practice
with the strongest evidence of effectiveness is inclusion of mechanisms to prompt students to
reflect on their level of understanding as they are learning online. In a related vein, there is some
evidence that online learning environments with the capacity to individualize instruction to a
learner’s specific needs improves effectiveness.
As noted in chapter 4, the results of studies using purely online and blended conditions cast some
doubt on the meta-analysis finding of larger effect sizes for studies blending online and face-to-
face elements. The inconsistency in the implications of the two sets of studies underscores the
importance of recognizing the confounding of practice variables in most studies. Studies using
blended learning also tend to involve more learning time, additional instructional resources, and
course elements that encourage interactions among learners. This confounding leaves open the
possibility that one or all of these other practice variables, rather than the blending of online and
offline media per se, accounts for the particularly positive outcomes for blended learning in the
studies included in the meta-analysis.
Comparison With Meta-Analyses of Distance Learning
Because online learning has much in common with distance learning, it is useful to compare the
findings of the present meta-analysis with the most comprehensive recent meta-analyses in the
distance-learning field. The two most pertinent earlier works are those by Bernard et al. (2004)
and Zhao et al. (2005). As noted above, the corpus in this meta-analysis differed from the earlier
quantitative syntheses, not only in including more recent studies but also in excluding studies
that did not involve Web-based instruction and studies that did not examine an objective student
learning outcome.
Bernard et al. (2004) found advantages for asynchronous over synchronous distance education, a
finding that on the surface appears incongruent with the results reported here. On closer
inspection, however, it turns out that the synchronous distance-education studies in the Bernard
et al. corpus were mostly cases of a satellite classroom yoked to the main classroom where the
52

54

performing a second literature search with an expanded time frame (through July 2008), the team
was able to greatly expand the corpus of studies with controlled designs and to identify five
controlled studies of K–12 online learning with seven contrasts between online and face-to-face
conditions. This expanded corpus still comprises a very small number of studies, especially
considering the extent to which secondary schools are using online courses and the rapid growth
of online instruction in K–12 education as a whole. Educators making decisions about online
learning need rigorous research examining the effectiveness of online learning for different types
of students and subject matter as well as studies of the relative effectiveness of different online
learning practices.

databases of raw student performance data and did not describe learning conditions, technology use or
learner/instructor characteristics. A recent large-scale study by the Florida TaxWatch (2007) failed to control for
preexisting differences between the students taking courses online and those taking them in conventional
classrooms.

eBiostat Solutions. 2006. Comprehensive Meta-Analysis (Version 2.2.027). [Computer software].
Mt. Airy, Md.: Biostat Solutions.
bBixler, B. A. 2008. The effects of scaffolding student’s problem-solving process via question
prompts on problem solving and intrinsic motivation in an online learning environment. PhD
diss., The Pennsylvania State University, State College, Penn.
eBransford, J. D., A. L. Brown, and R. R. Cocking. 1999. How people learn: Brain, mind,
experience, and school. Washington, D.C.: National Academy Press.
dBrown, B. A., and K. Ryoo. 2008. Teaching science as a language: A “content-first” approach
to science teaching. Journal of Research in Science Teaching 45 (5):529–53.
b, cCaldwell, E. R. 2006. A comparative study of three instructional modalities in a computer
programming course: Traditional instruction, Web-based instruction, and online instruction.
PhD diss., University of North Carolina at Greensboro.
bCampbell, M., W. Gibson, A. Hall, D. Richards, and P. Callery. 2008. Online vs. face-to-face
discussion in a Web-based research methods course for postgraduate nursing students: A
quasi-experimental study. International Journal of Nursing Studies 45 (5):750–59.
bCastaneda, R. 2008. The impact of computer-based simulation within an instructional sequence
on learner performance in a Web-based environment. PhD diss., Arizona State University,
Tempe.
eCavanaugh, C. 2001. The effectiveness of interactive distance education technologies in K–12
learning: A meta-analysis. International Journal of Educational Telecommunications 7
(1):73–78.
eCavanaugh, C., K. J. Gillan, J. Kromrey, M. Hess, and R. Blomeyer. 2004. The effects of
distance education on K–12 student outcomes: A meta-analysis. Naperville, Ill.: Learning
Point Associates. http://www.ncrel.org/tech/distance/index.html (accessed March 5, 2009).
b, cCavus, N., H. Uzonboylu, and D. Ibrahim. 2007. Assessing the success rate of students using a
learning management system together with a collaborative tool in Web-based teaching of
programming languages. Journal of Educational Computing Research 36 (3):301–21.
eChang, C. C. 2008. Enhancing self-perceived effects using Web-based portfolio assessment.
Computers in Human Behavior 24 (4):1753–71.
bChang, M. M. 2007. Enhancing Web-based language learning through self-monitoring. Journal
of Computer Assisted Learning 23 (3):187–96.
bChen, B. 2007. Effects of advance organizers on learning and retention from a fully Web-based
class. PhD diss., University of Central Florida, Orlando.
eChilds, J. M. 2001. Digital skill training research: Preliminary guidelines for distributed
learning (Final report). Albuquerque, N. Mex.: TRW. Available through
http://www.stormingmedia.us/24/2471/A247193.html (accessed March 5, 2009).
bChoi, I., S. M. Land, and A. J. Turgeon. 2005. Scaffolding peer-questioning strategies to
facilitate metacognition during online small group discussion. Instructional Science 33 (5–
6):483–511.
56

bChung, S., M.-J. Chung, and C. Severance. 1999, October. Design of support tools and
knowledge building in a virtual university course: Effect of reflection and self-explanation
prompts. Paper presented at the WebNet 99 World Conference on the WWW and Internet
Proceedings, Honolulu, Hawaii. (ERIC Document Reproduction Service No. ED448706).
eClark, R. E. 1983. Reconsidering research on learning from media. Review of Educational
Research 53 (4):445–49.
eClark, R. E. 1994. Media will never influence learning. Educational Technology Research and
Development 42 (2):21–29.
Cohen, J. 1992. A power primer. Psychological Bulletin, 112: 155–159.
bCook, D. A., D. M. Dupras, W. G. Thompson, and V. S. Pankratz. 2005. Web-based learning in
residents’ continuity clinics: A randomized, controlled trial. Academic Medicine 80 (1):90–
97.
bCook, D. A., M. H. Gelula, D. M. Dupras, and A. Schwartz. 2007. Instructional methods and
cognitive and learning styles in Web-based learning: Report of two randomised trials.
Medical Education 41 (9):897–905.
bCrippen, K. J., and B. L. Earl. 2007. The impact of Web-based worked examples and self-
explanation on performance, problem solving, and self-efficacy. Computers & Education 49
(3):809–21.
b, cDavis, J. D., M. Odell, J. Abbitt, and D. Amos. 1999, March. Developing online courses: A
comparison of Web-based instruction with traditional instruction. Paper presented at the
Society for Information Technology & Teacher Education International Conference,
Chesapeake, Va.
http://www.editlib.org/INDEX.CFM?fuseaction=Reader.ViewAbstract&paper_id=7520
(accessed March 5, 2009).
aDay, T. M., M. R. Raven, and M. E. Newman. 1998. The effects of World Wide Web
instruction and traditional instruction and learning styles on achievement and changes in
student attitudes in a technical writing in agricommunication course. Journal of Agricultural
Education 39 (4):65–75.
aDeBord, K. A., M. S. Aruguete, and J. Muhlig. 2004. Are computer-assisted teaching methods
effective? Teaching of Psychology 31 (1):65–68.
eDede, C., ed. 2006. Online professional development for teachers: Emerging models and
methods. Cambridge, Mass.: Harvard Education Publishing Group.
bDe Wever, B., M. Van Winckel, and M. Valcke 2008. Discussing patient management online:
The impact of roles on knowledge construction for students interning at the paediatric ward.
Advances in Health Sciences Education 13 (1):25–42.
bDinov, I. D., J. Sanchez, and N. Christou. 2008. Pedagogical utilization and assessment of the
statistic online computational resource in introductory probability and statistics courses.
Computers & Education 50 (1):284–300.
dDuphorne, P. L., and C. N. Gunawardena. 2005. The effect of three computer conferencing
designs on critical thinking skills of nursing students. American Journal of Distance
Education 19 (1):37–50.
57

aEl-Deghaidy, H., and A. Nouby. 2008. Effectiveness of a blended e-learning cooperative
approach in an Egyptian teacher education programme. Computers & Education 51 (3):988–
1006.
aEnglert, C. S., Y. Zhao, K. Dunsmore, N. Y. Collings, and K. Wolbers. 2007. Scaffolding the
writing of students with disabilities through procedural facilitation: Using an Internet-based
technology to improve performance. Learning Disability Quarterly 30 (1):9–29.
bEvans, K. L. 2007. Learning stoichiometry: A comparison of text and multimedia instructional
formats. PhD diss., University of Pittsburgh, Penn.
eFlorida TaxWatch. 2007. Final report: A comprehensive assessment of Florida virtual school.
Tallahassee, Fla.: Florida TaxWatch Center for Educational Performance and Accountability.
www.floridataxwatch.org/resources/pdf/110507FinalReportFLVS.pdf (accessed March 5,
2009).
dFox, E. J., and H. J. Sullivan. 2007. Comparing strategies for teaching abstract concepts in an
online tutorial. Journal of Educational Computing Research 37 (3):307–30.
aFrederickson, N., P. Reed, and V. Clifford. 2005. Evaluating Web-supported learning versus
lecture-based teaching: Quantitative and qualitative perspectives. Higher Education 50
(4):645–64.
bGaddis, B., H. Napierkowski, N. Guzman, and R. Muth. 2000, October. A comparison of
collaborative learning and audience awareness in two computer-mediated writing
environments. Paper presented at the National Convention of the Association for Educational
Communications and Technology, Denver, Colo. (ERIC Document Reproduction Service
No. ED455771).
bGao, T., and J. D. Lehman. 2003. The effects of different levels of interaction on the
achievement and motivational perceptions of college students in a Web-based learning
environment. Journal of Interactive Learning Research 14 (4):367–86.
aGilliver, R. S., B. Randall, and Y. M. Pok. 1998. Learning in cyberspace: Shaping the future.
Journal of Computer Assisted Learning 14 (3):212–22.
bGolanics, J. D., and E. M. Nussbaum. 2008. Enhancing online collaborative argumentation
through question elaboration and goal instructions. Journal of Computer Assisted Learning
24 (3):167–80.
bGrant, L. K., and M. Courtoreille. 2007. Comparison of fixed-item and response-sensitive
versions of an online tutorial. Psychological Record 57 (2):265–72.
dGriffin, T. J. 2007. Liberating the spacing effect from the laboratory: A practical application in a
worldwide Web-based religious education volunteer-teacher training program. PhD diss.,
Utah State University, Logan.
aHairston, N. R. 2007. Employees’ attitudes toward e-learning: Implications for policy in
industry environments. PhD diss., University of Arkansas, Fayetteville.
eHarlen, W., and S. Doubler. 2004. Can teachers learn through enquiry online? Studying
professional development in science delivered online and on-campus. International Journal
of Science Education 26 (10):1247–67.
58

bShih, Y. E. 2007. Dynamic language learning: Comparing mobile language learning with online
language learning. PhD diss., Capella University, Minneapolis, Minn..
eSitzmann, T., K. Kraiger, D. Stewart, and R. Wisher. 2006. The comparative effectiveness of
Web-based and classroom instruction: A meta-analysis. Personnel Psychology 59:623–64.
bSmith, C. M. 2006. Comparison of Web-based instructional design strategies in a pain
management program for nursing professional development. PhD diss., State University of
New York at Buffalo.
aSpires, H. A., C. Mason, C. Crissman, and A. Jackson. 2001. Exploring the academic self within
an electronic mail environment. Research and Teaching in Developmental Education 17
(2):5–14.
bStanley, O. L. 2006. A comparison of learning outcomes by ‘in-course’ evaluation techniques
for an online course in a controlled environment. The Journal of Educators Online 3 (2):1–
16.
eStevens, K. 1999. Two Canadian approaches to teaching biology, chemistry, mathematics and
physics to senior high school students in virtual classes. Paper presented at the Australasian
Science Education Research Association, Rotorua, New Zealand. (ERIC Document
Reproduction Service No. ED451987).
bSuh, S. 2006. The effect of using guided questions and collaborative groups for complex
problem solving on performance and attitude in a Web-enhanced learning environment. PhD
diss., Florida State University, Tallahassee.
aSun, K., Y. Lin, and C. Yu. 2008. A study on learning effect among different learning styles in a
Web-based lab of science for elementary school students. Computers & Education 50
(4):1411–22.
aSuter, W. N., and M. K. Perry. 1997. Evaluation by electronic mail. Paper presented at the
annual meeting of the Mid-South Educational Research Association, Memphis, Tenn.. (ERIC
Document Reproduction Service No. ED415269).
eTallent-Runnels, M. K., J. A. Thomas, W. Y. Lan, S. Cooper, T. C. Ahern, S. M. Shaw, and X.
Liu. 2006. Teaching courses online: A review of research. Review of Educational Research
76:93–135.
bTantrarungroj, P. 2008. Effect of embedded streaming video strategy in an online learning
environment on the learning of neuroscience. PhD diss., Indiana State University, Terre
Haute.
eTeague, G., and R. H. Riley. 2006. Does it improve high school students’ ability to perform
cardiopulmonary resuscitation in a simulated environment? Resuscitation 71 (3):352–57.
dTian, L., S. Tang, W. Cao, K. Zhang, V. Li, and R. Detels. 2007. Evaluation of a Web-based
intervention for improving HIV/AIDS knowledge in rural Yunnan, China. AIDS 21 (8):137–
42.
bTselios, N. K., N. M. Avouris, A. Dimitracopoulou, and S. Daskalaki. 2001. Evaluation of
distance-learning environments: Impact of usability on student performance. International
Journal of Educational Telecommunications 7 (4):355–78.
64

eTucker, B. 2007, June. Laboratories of reform: Virtual high schools and innovation in public
education. Washington, D.C.: Education Sector Reports.
http://www.butlertech.org/ek_sitepath/uploadedfiles/Teen_Education/Butler_Tech_Online/L
aboratories%20of%20Reform.pdf (accessed March 5, 2009).
aTurner, M. K., S. R. Simon, K. C. Facemyer, L. M. Newhall, and T. L. Veach. 2006. Web-based
learning versus standardized patients for teaching clinical diagnosis: A randomized,
controlled, crossover trial. Teaching and Learning in Medicine 18 (3):208–14.
aUrban, C. Q. 2006. The effects of using computer-based distance education for supplemental
instruction compared to traditional tutorial sessions to enhance learning for students at-risk
for academic difficulties. PhD diss., George Mason University, Fairfax, Va.
eUzunboylu, H. 2004. The effectiveness of Web assisted English language instruction on the
achievement and attitude of the students. In Proceedings of World Conference on
Educational Multimedia, Hypermedia and Telecommunications 2004, ed. L. Cantoni & C.
McLoughlin, 727–33. Chesapeake, Va.: Association for the Advancement of Computing in
Education. (ERIC Document Reproduction Service No. ED490528).
dVance, D., J. Dawson, V. Wadley, J. Edwards, D. Roenker, and M. Rizzo. 2007. The accelerate
study: The longitudinal effect of speed of processing training on cognitive performance of
older adults. Rehabilitation Psychology 52 (1):89–96.
aVandeweerd, J.-M. E. F., J. C. Davies, G. L. Pichbeck, and J. C. Cotton. 2007. Teaching
veterinary radiography by e-learning versus structured tutorial: A randomized, single-blinded
controlled trial. Journal of Veterinary Medical Education 34 (2):160–67.
eVrasidas, C., and G. V. Glass. 2004. Teacher professional development: Issues and trends. In
Online professional development for teachers, ed. C. Vrasidas and G. V. Glass, 1–12).
Greenwich, Conn.: Information Age.
aWallace, P. E., and R. B. Clariana. 2000. Achievement predictors for a computer-applications
module delivered online. Journal of Information Systems Education 11 (1/2):13–18.
aWang, L. 2008. Developing and evaluating an interactive multimedia instructional tool:
Learning outcomes and user experiences of optometry students. Journal of Educational
Multimedia and Hypermedia 17 (1):43–57.
bWang, K. H., T. H. Wang, W. L. Wang, and S. C. Huang. 2006. Learning styles and formative
assessment strategy: Enhancing student achievement in Web-based learning. Journal of
Computer Assisted Learning 22 (3):207–17.
bWeinberger, A., B. Ertl, F. Fischer, and H. Mandl. 2005. Epistemic and social scripts in
computer supported collaborative learning. Instructional Science 33 (1):1–30. [Two studies
are reported.]
eWestEd with Edvance Research. 2008. Evaluating online learning: Challenges and strategies for
success. Washington, D.C.: U.S. Department of Education. http://evalonline.ed.gov/
(accessed March 5, 2009).
eWhat Works Clearinghouse. 2007. Technical details of WWC-conducted computations.
Washington, D.C.: U.S. Department of Education.
http://ies.ed.gov/ncee/wwc/pdf/conducted_computations.pdf (accessed March 5, 2009).
65

Appendix
Meta-Analysis Methodology
Terms and Processes Used in the Database Searches
In March 2007, researchers performed searches through the following four data sources:
1. Electronic research databases. Using a common set of keywords (see Exhibit A-1),
searches were performed in ERIC, PsycINFO, PubMed, ABI/INFORM, and UMI
ProQuest Digital Dissertations. In addition, to make sure that studies of online
learning in teacher professional development and career technical education were
included, additional sets of keywords, shown in Exhibit A-2, were used in additional
searches of ERIC and PsycINFO.
2. Recent meta-analyses and narrative syntheses. Researchers reviewed the lists of
studies included in Bernard et al. (2004), Cavanaugh et al. (2004), Childs (2001),
Sitzmann et al. (2006), Tallent-Runnels et al. (2006), Wisher and Olson (2003), and
Zhao et al. (2005) for possible inclusions. Additionally, for teacher professional
development and career technical education, references from recent narrative research
syntheses in those fields (Whitehouse et al. 2006; Zirkle 2003) were examined to
identify potential studies for inclusion.
3. Key journals. Abstracts were manually reviewed for articles published since 2005 in
American Journal of Distance Education, Journal of Distance Education (Canada),
Distance Education (Australia), International Review of Research in Distance and
Open Education, and Journal of Asynchronous Learning Networks. In addition, the
Journal of Technology and Teacher Education and Career and Technical Education
Research (formerly known as Journal of Vocational Education Research) were
manually searched.
4. Google Scholar searches. To complement these targeted searches, researchers used
limiting parameters and sets of keywords (available from the authors of this report) in
the Google Scholar search engine.

A-1

Exhibit A-1. Terms for Initial Research Database Search
Technology and
Education/
Training Terms

Study Design Termsa
Distance education Control group
Distance learning Comparison group
E-learning Treatment group
Online education Experimental
Online learning
Online training
Online course
Virtual learning
Virtual training
Virtual & course
Internet & learning
Internet & training
Internet & course
Web-based learning
Web-based instruction
Web-based course
Web-based training
“Distributed learning”
a All four terms were used in one query with “OR” if the
database allowed.


Exhibit A-2. Terms for Additional Database Searches for Online Career Technical Education and
Teacher Professional Development
Education Terms
Technology
Terms Study Design Terms
Career education Distance Control group
Vocational education Distributed Comparison group
Teacher education E-learning Experimental
Teacher mentoring Internet Randomized
Teacher professional
development
Online Treatment group
Teacher training Virtual
Technical education Web-based

A-2

• In general, one learning outcome finding was extracted from each study. When multiple
learning outcome data were reported (e.g., assignments, midterm and final examinations,
grade point averages, grade distributions), the outcome that could be expected to be more
stable and more closely aligned to the instruction was extracted (e.g., final examination
scores instead of quizzes). However, in some studies, no learning outcome had obvious
superiority over the others. In such cases, analysts extracted multiple contrasts from the
study and calculated the weighted average of the multiple outcome scores if the outcome
measures were similar (e.g., two final tests, one testing procedural skills and the other
testing declarative knowledge). For example, in one study, analysts retained two outcome
findings because the outcome measures were quite different (Schilling et al. 2006). One
measure was a multiple-choice test, examining basic knowledge, whereas the other was a
performance-based assessment, testing students’ strategic and problem-solving skills in
the context of ill-structured problems.
• Learning outcome findings were extracted at the individual level. Analysts did not extract
group-level learning outcomes (e.g., scores for a group product). Too few group products
were included in the studies to support analyses of this variable.
The review of the 99 studies for effect size calculation produced 51 independent effect sizes (28
for Category 1 and 23 for Category 2) from 46 studies; 53 studies did not report sufficient data to
support effect-size calculation.
Coding of Study Features
All studies that provided enough effect size data were coded for their study features and for study
quality. The top-level coding structure, incorporating refinements made after pilot testing, is
shown in Exhibit A-4. (The full coding structure is available from the authors of this report.)
Twenty percent of the studies with sufficient data to compute effect size were coded by two
researchers. The interrater reliability across these double-coded studies was 86.4 percent. As a
result of analyzing coder disagreements, some definitions and decision rules for some codes were
refined; other codes that required information missing in the vast majority of documents or that
proved difficult to code reliably (e.g., indication of whether the instructor was certified or not)
were eliminated. A single researcher coded the remaining studies.









A-5

Exhibit A-4. Top-level Coding Structure for the Meta-analysis
Study Feature Coding Categories
• Study type
• Type of publication
• Year of publication
• Study author
• Whether the instructor was trained in online training
• Learner type
• Learner age
• Learner incentive for involvement in the study
• Learning setting
• Subject matter
• Treatment duration
• Dominant approach to learner control
• Media features
• Opportunity for face-to-face contact with the instructor
• Opportunity for face-to-face contact with peers
• Opportunity for asynchronous computer-mediated communication with the instructor
• Opportunity for asynchronous computer-mediated communication with peers
• Opportunity for synchronous computer-mediated communication with the instructor
• Opportunity for synchronous computer-mediated communication with peers
• Use of problem-based or project-based learning
• Opportunity for practice
• Opportunity for feedback
• Type of media-supported pedagogy
• Nature of outcome measure
• Nature of knowledge assessed
Study Design Codes
• Unit of assignment to conditions
• Sample size for unit of assignment
• Student equivalence
• Whether equivalence of groups at preintervention was described
• Equivalence of prior knowledge/pretest scores
• Instructor equivalence
• Time-on-task equivalence
• Curriculum material/instruction equivalence
• Attrition equivalence
• Contamination
A-6