Evaluating Trustworthiness in BYOD Online Exams via Smartphone Screenshots

Abstract

Screenshots serve as valuable data, offering visual insights into user behavior. Examiners can use screenshots in online exams to track the various activities they conduct on their computing devices. This study attempts to identify various potential dishonest behaviors that may be exhibited by participants in bring your own device (BYOD) online exams using smartphones. Initially, we designed an automated screen capture model to capture screenshots on participants’ devices. The system is modeled using system modeling language (SysML), encompassing the component structure as well as the functional and non-functional requirements. Three alternative algorithms were proposed: periodic timers at set intervals, event-driven triggers based on user interface changes, and a combination of both. The system was successfully implemented on an Android-based Learning Management System (LMS), which is commonly used for examinations in educational institutions. The verification involved conducting series of tests on three Android devices and assessing its performance using Android Studio’s profiling features. Performance measurements of 9\times 2 alternative configurations of the proposed screen capture algorithms in data collection and inference modes were analyzed using the Technique for Order Preference by Similarity to Ideal Solution (TOPSIS) to identify the optimal configuration. The analysis concluded that the best algorithm configuration for both modes, with a preference value of 0.6, is the periodic timer with a five-second interval configuration. The system was validated through two simulated exam sessions involving 45 student participants who also represented the target users. Each participant used their Android device, resulting in 45 unique devices being tested. This setup allows the evaluation of the system’s performance and reliability across a diverse range of hardware and user environments. Their perceptions of the system’s usability were assessed using the Usability Metric for User Experience (UMUX-LITEr) questionnaire to ensure that the system was easy to use and met their requirements. The results indicated that the system’s usability received grade A, with a score of 80.8. This validation process also resulted in a dataset of 4,510 screenshots that were manually labeled to identify various cheating behaviors (17 potential dishonest behaviors). Participants most frequently performed three dishonest actions: searching for answers using search engines, opening webpages, and asking AI assistants.