Driver Monitoring System Using Computer Vision for Real-Time Detection of Fatigue, Distraction and Emotion via Facial Landmarks and Deep Learning

Abstract

Highlights: What are the main findings? The system detects drowsiness using the EAR and the MAR with high accuracy (88.89% and 85.19%, respectively). Distraction behaviors based on head pose and gaze are detected with 100% accuracy. Emotion recognition performs well for happiness (100%), anger/disgust (96.3%), and surprise (92.6%), while the detection of sadness (66.7%) and fear (0%) remains more challenging due to atypical real-world expressions. The system operates in real time using a convolutional neural network based on MobileNetV2 and facial landmarks from MediaPipe. What are the implications of the main findings? The real-time detection of fatigue and distraction can improve road safety and prevent accidents. Including emotion recognition enhances detection accuracy by providing context to facial cues. The use of non-invasive, efficient methods allows deployment in real driving environments. Identifying emotional and attentional states can support the development of smarter, adaptive vehicle safety systems. Car accidents remain a leading cause of death worldwide, with drowsiness and distraction accounting for roughly 25% of fatal crashes in Ecuador. This study presents a real-time driver monitoring system that uses computer vision and deep learning to detect fatigue, distraction, and emotions from facial expressions. It combines a MobileNetV2-based CNN trained on RAF-DB for emotion recognition and MediaPipe’s 468 facial landmarks to compute the EAR (Eye Aspect Ratio), the MAR (Mouth Aspect Ratio), the gaze, and the head pose. Tests with 27 participants in both real and simulated driving environments showed strong results. There was a 100% accuracy in detecting distraction, 85.19% for yawning, and 88.89% for eye closure. The system also effectively recognized happiness (100%) and anger/disgust (96.3%). However, it struggled with sadness and failed to detect fear, likely due to the subtlety of real-world expressions and limitations in the training dataset. Despite these challenges, the results highlight the importance of integrating emotional awareness into driver monitoring systems, which helps reduce false alarms and improve response accuracy. This work supports the development of lightweight, non-invasive technologies that enhance driving safety through intelligent behavior analysis.