Author's persona! copy EDUCATIONAL EVALUATION - CONTEMPORARY THEMES IN EVALUATION Contents Evaluating ENLearning Evaluating Schools as Learning Communities Evaluation of Adult Education and Training Programs Evaluation of Integrated Health Programs in School Formative Assessment in Teacher Education and Teacher Professional Development Informal Education and Evaluation Methods to Evaluate Technology Evaluating E-Learning D D Williams and C R Graham, Brigham Young University, Provo, UT, USA �� 2010 Elsevier Ltd. All rights reserved. Glossary Evaluating E-Learning Hopeful visions for using new technologies are trans- forming the nature of learning and possibilities for 530 teaching people of all ages (Wesch, 2007). :Multiple par- ticipants' innovations with technology-mediated learn- ing require dynamic evaluative inquiry (Allen and Seam,ln, 2007 "VestEd, 2008). Evaluators identify evol- ving evaluation objects (evaluands) in idiosyncratic com- binations, involve stakeholclers in prioritizing their values to select from thousands of possible evaluation questions, gather, analyze, and report answers to those questions, and make recommendations for action, while simulta- neously meta-evaluating and improving their evaluation activities. This article employs a flexible evaluation framework (Williams, 2006), exemplified in most articles of this volume and in evaluation literature, to illustrate a way to conduct e-Iearning evaluations (see Figure 1). Consideration of challenges and opportunities facing e-Iearning participants and evaluators, as educational uses oftechnologies proliferate, complete the article. As Figure 1 illustrates, evaluation involves several activities for reaching judgments by clarifying what should be. This usually entails understanding the evalu- ation context by consulting with stakeholders to identify evaluands to evaluate, clarify criteria that stakeholders value, and compose questions about the evaluands' nature and performance. Then, evaluators collaborate with stakeholders to determine what is by clarifying methods to effectively answer the questions, and collect- ing and analyzing associated data. Finally, evaluators and stakeholders juxtapose what should be with what is to judge how well what is meets what should be, make recommendations, and meta-evaluate the entire process and results. International Encyclopedia of Education (2010), vol. 3, pp. 530-538

Evaluating E..Learning 531 �� What Is the evaluand? @ What are the criteria for judging the evaluand? G) What is the context/background? �� Who are the stakeholders? What should be Determine ,/ '" ~~~~~~~~~-1 """""'""", r--~~~~~~~~-"""'/:.....~--/_/-/-/~~ �� What questions will answer how well the evaluand meets the criteria? What is Determine \ �� What methods should be used to answer the questions? (J) Collect and analyze data �� Evaluation: How does what is compare to what should be? �� Reporting: What recommendations does the study yield? @ Metaevaluatlon: How well was the evaluation conducted? Figure 1 Key components in the evaluation process. Table 1 Scope, stakeholders, and possible evaluands in an e-Iearning context Scope of context Primary stakeholders Sample evaluands Example criteria Institutional level Program level Course level Activity level Administrators Administrators Instructor/learners Instructor/learners E-Iearning initiative Online course offerings E-learning policies Distance-learning program Online course Course activity Cost effectiveness Enrolments Completion rates/satisfaction Enrolments Student satisfaction Student satisfaction, learning, engagement Student access Resource/tech requirements Student satisfaction, learning, engagement Participant time involvement Elements of E-Learning Evaluation After illustrating how activities summarized in Figure 1 facilitate evaluation of e-Iearning experiences and results, an extended case is presented to demonstrate processes for clarifying what should be, describing what is, and comparing them. Clarifying What Should Be Clarifying what should be involves identifJring context, stakeholders, evaluands, and criteria to generate evaluation questions. Context Beginning an e-learning evaluation involves understand- ing the context and setting the stage for deciding who the stakeholders will be, what evaluands ,vill be studied, and what the criteria for judging the evaluands will be. For example, Table 1 shov,rs four scope levels that ate often consideted in e-Iearning contexts (Graham, 2006). At the institutional and progmm levels, administrators are often the primary stakeholders, while at the course and activity levels, faculty and learners are often the primary stakeholders. Context also includes other variables summarized in Table 2: The setting for e-lea1'l1ing, the evaluation purpose, International Enoyclopedia of Eduoation (2010), vol. 3, pp. 530-538