Fatigue driving detection based on ocular self-quotient image and gradient image co-occurrence matrix

Abstract

Objective: Driver fatigue is known to be directly related to road safety and is a leading cause of traffic fatalities and injuries of seated drivers. Previous studies used many fatigue driving detection methods to detect and analyze the fatigue status of seated drivers. These methods aim to improve detection accuracy and usually include driving behavioral features (e.g., steering wheel motion, lane keeping) and physiological features (e.g., eye and face movement, heart rate variability, electroencephalogram, electroocoulogram, electrocardiogram). Physiological features, such as eye movement, are widely used to predict driver fatigue because they are nonintrusive and independent on the driving context. However, fatigue driving detection under occluded face conditions is challenging and needs a robust algorithm of eye feature extraction. The literature showed that most eye tracking methods require high-resolution images. This condition leads to low processing speed and difficultly on real-time eye tracking. In this study, a fatigue driving detection method based on self-quotient image (SQI) and gradient image co-occurrence matrix was presented. This improved method is based on gray level and gradient co-occurrence matrix. The proposed method provides a new approach for predicting fatigue status on driver fatigue applications in a short time. Method: In this study, a six-degree of freedom vibration table and driving simulator were used to model the driving context. The eye fatigued state of seated driver in real time was recorded by using an RGB camera mounted in the front of the driver. A single shot multibox detector face detection algorithm was used to extract the driver's facial region from the recorded video with ResNet10 as the front network. An ensemble of regression trees facial landmark location algorithm was used to calibrate the driver's eye area for each frame of the recorded video. A gray-level image and an SQI of each frame was combined to obtain the co-occurrence matrix of the driver's eye image. The statistic of numerical characteristics of the SQI and gradient image co-occurrence matrix of driver's eye images was analyzed. The driver's eye states were determined in accordance with the statistical features of the matrix and had a large variation. Percentage of eyelid closure(PERCLOS) over the pupil over time and maximum closing duration(MCD) were utilized to predict the state transition of seated driver from nonfatigue to fatigue. Result: We compared our method with other fatigue driving detection methods based on eye aspect ratio and based on convolutional neural network (CNN) model. The quantitative evaluation metrics contained face detection accuracy, detection accuracy of open and closed states of the eye, and blink detection accuracy. The experiment results show that our model has better performance than other methods in video datasets. These datasets were captured in the driving context modeled by six-degree of freedom vibration table and driving simulator. The following cases were studied to predict the fatigue status of seated driver. One case was detected under occluded face conditions (e.g., wearing a mask and wearing glass), and the other case had no mask or glass. The accuracy of eye state analysis of method 1 is 97.68%, but the accuracy depends on the positioning accuracy of facial landmarks. However, the overall detection speed of the algorithm is slow and sensitive to whether the driver wore a mask. This condition makes the detection of the eye state module invalid. Classifying the driver eye state with the CNN takes a long time, and the eye state accuracy of method 2 is more than 96%. However, the success rate of blink detection is approximately zero for hard video samples. As a comparison, the proposed algorithm has better performance in face detection, eye opening and closing, and blink accuracies. The accuracy of opening and closing the eyes reaches 98.73% for a video sample of drivers' face under occlusion, such as wearing a mask. The recognition precision is 99.52% for a video sample of drivers without mask or glass, and the video sample processing frame rate is up to 32 frame/s. Conclusion: In this study, we proposed a fatigue driving detection method based on SQI and gradient image co-occurrence matrix. The experiment results demonstrate that our method has better performance than other several fatigue driving detection methods and can detect effectively the driver's eyes state when they open and close in real time. The proposed method has high accuracy and processing speed.