How Far Can We Trust the Predictions of Learning Analytics Systems?

Abstract

Prediction systems based on Machine Learning (ML) models for teachers are widely used in the Learning Analytics (LA) field to address the problem of high failure rates in online learning. One objective of these systems is to identify at-risk of failure learners so that teachers can intervene effectively with them. Therefore, teachers’ trust in the reliability of the predictive performance of these systems is of great importance. However, despite the relevance of this notion of trust, the literature does not propose particular methods to measure the trust to be granted to the system results. In this paper, we develop an approach to measure a teacher’s trust in the prediction accuracy of an LA system. For this aim, we define three trust granularities, including: the overall trust, trust per class label and trust per prediction. For each trust granularity, we proceed to the calculation of a Trust Index (TI) using the concepts of confidence level and confidence interval of statistics. As a proof of concept, we apply this approach on a system using the Random Forest (RF) model and real data of online k-12 learners.