Research on a Lightweight Fatigue Detection Method for Drivers Based on Multimodal Feature Fusion

Abstract

Driver fatigue is a leading cause of traffic accidents, making its accurate detection highly significant for improving road safety. Traditional detection methods often suffer from poor stability in complex environments and reliance on limited features. To overcome these limitations, this study proposes a lightweight multimodal deep learning model that integrates visual characteristics with physiological indicators. The model enhances detection accuracy by combining image features with physiological behavioral features. To address the inadequate adaptability of traditional single-modal methods in complex scenarios, a dual-branch architecture integrating a Convolutional Neural Network (CNN) and handcrafted features was designed: the image branch employs a three-layer convolutional structure to extract deep facial visual features, while the feature branch incorporates the Eye Aspect Ratio (EAR) and Mouth Aspect Ratio (MAR) as key physiological indicators. To tackle the issue of imbalanced data distribution, a dynamic class weighting strategy was applied to optimize the loss function. Furthermore, L2 regularization and Dropout layers were utilized to suppress model overfitting, alongside StandardScaler normalization to enhance feature generalization capability. Experimental results demonstrate that the proposed method achieved an accuracy of 83% and an F1-score of 0.89 on the test set, significantly outperforming both single-feature models and traditional machine learning approaches. This effectively addresses the challenges of high false alarm rates and insufficient robustness in fatigue state detection.