Probability of detection (POD)-based metric for evaluation of classifiers used in driving behavior prediction

Abstract

Classifiers are functional tools/algorithms that implement classifications and are widely used in science and technology for state of health estimation, diagnosis systems, and situation/intention recognition of human operators. Certification of these classifiers plays a crucial role in their selection for a specific task. Current certification approaches utilize the Receiver Operator Curve (ROC) as a standard tool that provides graphically the performance of classifiers. Beside the ratio of Detection Rate and False Alarm Rate (combined as ROC), other properties related to process parameters are not considered. In this research, a new evaluation method based on the Probability of Detection (POD) reliability measure is developed discussing the effect of further process parameters on the classification results. Probability of Detection (POD) serves as a performance measure for quantifying the reliability of conventional Nondestructive Testing (NDT) procedures and Structural Health Monitoring (SHM) systems. The approach considers statistical variability of sensor-based measurements. In this publication for the first time the signal-response and the binary (hit/miss) approaches are implemented in combination with a process parameter. As example, the prediction time of driving behavior classification is used as process parameter. The signal-response approach is applied to compare Fuzzy Logic-Hidden Markov Model (FL-HMM), Artificial Neural Network (ANN), and Support Vector Machine (SVM) with respect to the reliability of the prediction time for driver behavior. The hit/miss approach is also applied on FL-HMM as example for predicting an upcoming driving maneuver. To account for data uncertainty and variability, confidence bounds are established. A typical and useful criteria for detection at a 90 % probability of detection level with 95 % confidence level is successfully implemented as a new reliability measure and certification standard for classifiers. In this article a new approach is established permitting a new evaluation measure to Machine Learning approaches used as classifiers. The proposed approach introduces a POD-based measure for comparison of binary classifiers.