Multimodal Assessment of Interest Levels in Reading: Integrating Eye-Tracking and Physiological Sensing

Abstract

Interest in the context of reading holds special significance as it serves as a driving force for learning and education. By understanding and leveraging students' interests, educators can create more effective and enjoyable learning environments that promote personalized learning experiences, enhanced comprehension, deep understanding, and motivation. A multimodal approach integrating gaze and physiological data could provide a more comprehensive and accurate assessment of interest levels. The goal of this study is to measure the level of interest experienced by users when reading newspaper articles by integrating gaze data and physiological responses. An experiment was conducted which recorded the gaze and physiological data from 13 university students reading 18 newspaper articles collected from the BBC news database. An SMI eye-tracker and an Empatica E4 wristband were used synchronously to capture the user's eye movements and physiological data. To predict the interest levels of the participants, a manual feature extraction-based approach and a deep learning-based approach were employed. The interest levels were divided into four-class and binary based on the responses from the participants. A CNN-LSTM model using the gaze features outperformed other models in terms of accuracy and F1-score with 52.8% and 51.8 for four-class and 82.3% and 81.7 for binary classification using leave-one-document-out and leave-one-participant-out cross-validation, respectively.