A multimodal framework for fatigue driving detection via feature fusion of vision and tactile information

Abstract

Driver fatigue is a major cause of traffic accidents, significantly impairing attention and reaction time. Traditional detection methods typically rely either on visual data or sensor signals. Image-based approaches suffer from lighting variations, while sensor-based methods are prone to noise interference. Here, a multimodal fusion architecture that integrates visual imagery with tactile signals from flexible sensors using porous composites is proposed to detect driver fatigue states. A convolutional neural network extracts features from the images, while sensor signals are encoded through fully connected layers. The extracted representations are then projected into the same dimensional space for concatenated feature fusion. Experimental results show that the proposed multimodal approach improves recognition accuracy by more than 4% compared with single-modality methods and generalizes reliably across varied appearances and extreme conditions. Furthermore, a detailed evaluation and quantification of fatigue levels have been conducted, which contributes to accident prevention and promotes driving safety.