Advanced POD-Based Performance Evaluation of Classifiers Applied to Human Driver Lane Changing Prediction

TL;DR

The paper tackles the challenge of evaluating ML classifiers for predicting human driver lane changes under a task-specific process parameter—the time remaining until the lane-change event—by adopting a probability-of-detection (POD) framework. It introduces a modified hit/miss POD that uses probabilistic ML outputs at each time step, aligning the POD evaluation more closely with a conventional $a$ versus $a$ approach while preserving simplicity. The methodology is applied to multiple classifier families (ANN, SVM, HMM, RF) with features learned from a deep autoencoder, using GA-optimized hyperparameters and a winner-take-all ensemble to compare performance. Results show that the modified POD yields $a_{90/95}$ values that are more consistent with the $a$ versus $a$ approach and generally lower than the standard POD, particularly for reliable early prediction, highlighting improved reliability for feature selection and deployment in safety-critical driving systems.

Abstract

Machine learning (ML) classifiers serve as essential tools facilitating classification and prediction across various domains. The performance of these algorithms should be known to ensure their reliable application. In certain fields, receiver operating characteristic and precision-recall curves are frequently employed to assess machine learning algorithms without accounting for the impact of process parameters. However, it may be essential to evaluate the performance of these algorithms in relation to such parameters. As a performance evaluation metric capable of considering the effects of process parameters, this paper uses a modified probability of detection (POD) approach to assess the reliability of ML-based algorithms. As an example, the POD-based approach is employed to assess ML models used for predicting the lane changing behavior of a vehicle driver. The time remaining to the predicted (and therefore unknown) lane changing event is considered as process parameter. The hit/miss approach to POD is taken here and modified by considering the probability of lane changing derived from ML algorithms at each time step, and obtaining the final result of the analysis accordingly. This improves the reliability of results compared to the standard hit/miss approach, which considers the outcome of the classifiers as either 0 or 1, while also simplifying evaluation compared to the â versus a approach. Performance evaluation results of the proposed approach are compared with those obtained with the standard hit/miss approach and a pre-developed â versus a approach to validate the effectiveness of the proposed method. The comparison shows that this method provides an averaging conservative behavior with the advantage of enhancing the reliability of the hit/miss approach to POD while retaining its simplicity.

Figures (4)

• Figure 1: Example of a POD curve obtained from hit/miss data, its 95 % lower confidence bound, and corresponding $a_{90}$ and $a_{90/95}$ values.
• Figure 2: SCANeR™ studio, Chair of Dynamics and Control, University of Duisburg–Essen, Germany.
• Figure 3: The driving environment and lane changing/keeping behaviors.
• Figure 4: Probability of detection curves obtained from the standard hit/miss approach for (a) the ANN and (b) the SVM trained on features from the first encoder layer to predict LCL using test data from the first driver.