Perceived Level of Usability as an Evaluation Metric in Adaptive E-learning

Abstract

The global spread of COVID-19 has shifted the learning process towards e-learning. In this context, a critical challenge for researchers is to understand and evaluate the effectiveness of e-learning, especially when the learning is adapted to the needs of individual users. In this work we argue that the learner’s perception of the level of usability of a system is a valuable metric that gives an insight into the learners’ engagement and motivation to learn. Little attention has been paid to this metric. In this paper we explore why this is important and valuable. We present a case study which uses the System Usability Scale (SUS) questionnaire to measure the user’s perception of usability as an indirect (proxy) measure of engagement. A between-subject experiment was conducted with 41 learners with dyslexia. The intervention group used the adaptive version of the e-learning system that matched the material to the needs of the learner. The control group used a standard version. At the end, learning gain and SUS scores were assessed. The correlation between learning performance and the perceived level of usability was positive and moderate (0.517, p < 0.05) among participants in the intervention group. However, learning performance and perceived level of usability were unrelated in the control group (− 0.364, p > 0.05). From this, and other work, it appears that using a learner’s assessment of the usability of a system is an effective way to measure their attitude to their learning. It reflects their perception of its suitability to their needs and this, in turn, is likely to affect their engagement and motivation. As such, this provides an effective instrument to judge whether adaptation based on learner needs has been successful.