Robust Driver State Assessment on an Embedded Platform Using a Fuzzy Belief Rule-Based Fusion of Eye Aspect Ratio and Voice Commands

Abstract

This research presents an Adaptive Driver Monitoring and Assistance System (ADMAS) built on a multimodal AI architecture to prevent accidents caused by critical driver states such as drowsiness, fatigue, and distraction. At its core, a Fuzzy Belief Rule-Based (FBRB) inference engine integrates multimodal data from camera and microphone sensors to ensure robust decision-making. The visual stream computes an innovative EAR4 feature derived from facial landmarks, achieving 89.4% recall in machine learning classification tasks. To enhance robustness, the system establishes a personalized adaptive threshold for each driver during an initialization phase by analyzing data from over 300 frames. Concurrently, a speech recognition module enables proactive, system-initiated assistance via a voice interface, allowing the system to deliver vocal prompts and manage functions based on conversational input, thereby minimizing driver distraction. The FBRB engine fuses the EAR-4 feature, head pose estimations, and voice intent to assess the driver’s state, subsequently triggering timely drowsiness alerts and autonomous system adjustments. The system, deployed on an NVIDIA Jetson Orin Nano for low-latency on-device processing, achieved reliable detection at 25 frames per second (FPS) under dynamic driving conditions, confirming its viability for real-world deployment.