An Action Research Approach to the Design, Development and Evaluation of an Interactive E-Learning Tutorial in a Cognitive Domain

Abstract

The teaching and learning of a complex section in "Theoretical Computer Science 1" in a distance-education context at the University of South Africa (UNISA) has been enhanced by a supplementary e-learning application called "Relations," which interactively teaches mathematical skills in a cognitive domain. It has tutorial and practice functionality in a classic computer-aided instruction (CAI) style and offers considerable learner control. A participative action research approach was used to design, develop, evaluate, and refine the application over a longitudinal period. In this process the application was formatively and summatively evaluated by different methods--questionnaire surveys, interviews, heuristic evaluation and a post-test. This article explains the purpose, structure, and operation of "Relations" and notes how the various evaluation methods resulted in iterative refinements to its functionality, learning content, and usability. The findings lead to reflection. Conventional computer-aided instruction and learning (CAI/L) has a role to play in the milieu of e-learning. CAI can present efficient instruction, motivate and engage learners, challenge them with meaningful exercises, and can support effective learning. The students requested more such tutorial and practice environments. "Relations'" greatest strength is its excellent diagnostic feedback, attested to by learners and expert evaluators alike. Courseware authoring systems have powerful facilities that can be used to judge the learner-input and provide appropriate, detailed, tailor-made feedback. This can be done in web-based learning (WBL) too, using specialized web-programming languages, but it is more complex. A further obstacle to the use of WBL at UNISA is that many UNISA students still lack broadband Internet access. The designer and developers of "Relations" used technology, not for its own sake, but rather to motivate and to illustrate concepts in ways that enhance cognition. Technology should be medium and not the message. The blue-water recreational theme was well received by the majority of learners and expert evaluators, who acknowledged its role in providing brief interludes of diversion and relaxation in a demanding cognitive domain. The concepts of usability and interaction design from the discipline of human-computer interaction (HCI) are receiving increasing attention in the development of commercial and corporate software. It is equally important to produce usable applications in educational contexts where the users are not professionals in the workplace, but learners who must first be able to use and interact with a system before they can even commence learning. Many learners approach e-learning after exposure to commercial software. As far as possible, learning applications should use operations and keypresses that support the HCI principles of predictability and consistency with familiar systems. In certain respects "Relations" falls short in this respect, but it was found to be easy to learn and use. It adheres to the fundamental principle of internal consistency, where a system's own internal operations are characterized by predictability and visibility. This study offers lessons for new CAI environments. The teaching approaches in "Relations" present multi-perspectives on a concept and multiple representations of content. These methods were successful and could be supplemented by multi-modal presentations in contexts where they would add value. There was a degree of learner-control, permitting choices about sections, sequence, repetition, and whether to omit theoretical segments and head straight for practice. The hyperlinks incorporated into "Relations V2.2" were a success; these hotspots provide direct access to definitions, elaborations and cross references. Concerns were raised regarding some aspects of control and navigation. Where pedagogically appropriate, linear sequences should be supplemented with hyperlinked access. Exit and re-entry facilities are required. User-control should be provided over the level of difficulty, so that users can choose particular exercises. This would facilitate different ways of usage at different times. The control structures deactivate certain buttons in particular situations. Although this was designed for appropriate contexts, some users objected. Ways of implementing such tactics must be carefully chosen, as designers exercise discernment in deciding when to provide the facilities that users request and when to overrule for pedagogical reasons. The multi-method evaluation of "Relations" had positive results. The complementary use of four evaluation techniques and triangulated data provided a synergic framework. Heuristic evaluations by experts were valuable sources of critique and suggestions. Two evaluators who are leading subject matter experts tested the feedback to its limits, revealing flaws and compromises of mathematical rigour. The experts moved beyond the given criteria, identifying omissions and suggesting further features. The discernment, insight, and long-standing expertise of the heuristic evaluators accurately pinpointed the strengths and weaknesses of "Relations", as well as subtle mathematical inaccuracies. For overall evaluation, the team of evaluators should include experts with subject matter skills and those with usability expertise. The students' questionnaire was particularly useful with regard to educational aspects. It confirmed the value of the detailed feedback, showing that the time spent painstakingly developing it was worthwhile. The 2004 questionnaire survey elicited particularly useful information and identified certain errors that were corrected even before "Relations" was presented to the heuristic evaluators. The inclusion of open-ended questions for qualitative responses served to elaborate responses, to enrich the information obtained, and to motivate the Likert options selected. In many cases, students used these to express praise and appreciation of "Relations". Unanticipated aspects that emerge can be probed further in interviews or subsequent questionnaires. The interviews among end-users (students) explored further avenues, following up on usability problems. The interviews probed for explicit identification of the features that fostered learning for each individual and the features they enjoyed most. UNISA's distance-teaching context makes contact difficult and the sample of volunteers used in this study represented a fairly homogenous group of young fulltime students, typical of the shift in learner profile. However, contact with a stereotypical group is informative in its own right. To optimize on interviews, the structure was flexible. The semi-structured interview was based on a set of common core questions, with follow-up questions prompted by the participants' responses. The post-tests gave quantitative measures that could be statistically analysed. However, scores in tests remain debatable instruments for measuring the effectiveness of an artefact or other intervention in improving learning. Tests frequently show no significant difference between methods of instruction, even when other evaluation methods generate favourable reports of the intervention. When a test, then, does indicate improved academic performance, it would appear that the method under investigation indeed enhances learning. This was the case in the present evaluation. The participative action research process of designing, developing, evaluating, and refining the e-learning tutorial "Relations" has taught the designer and development team a great deal--not just about the application being studied, but also lessons and generic principles for the design of e-learning applications as well as lessons about the rich complementary roles of different evaluation methods and techniques. (Contains 7 figures and 1 table.)