Towards Robust Unsupervised Attention Prediction in Autonomous Driving

TL;DR

The paper proposes a robust unsupervised framework for driving attention prediction that leverages multiple pseudo-labels, a Knowledge Embedding Block to inject traffic priors, and an Uncertainty Mining Branch to quantify cross-model uncertainty. It introduces RoboMixup, a pixel-level, attention-guided augmentation with random cropping to combat corruption and central bias, and a DriverAttention-C dataset to benchmark robustness under diverse degradations. Across BDD-A, DR(eye)VE, DADA-2000, and DriverAttention-C, the method achieves or surpasses fully supervised baselines and demonstrates reduced corruption degradation and improved central-bias robustness, validating its practicality for safety-critical driving tasks. The approach offers interpretable improvements in attention prediction and paves the way for robust, label-free self-driving perception in real-world, adversarial conditions.

Abstract

Robustly predicting attention regions of interest for self-driving systems is crucial for driving safety but presents significant challenges due to the labor-intensive nature of obtaining large-scale attention labels and the domain gap between self-driving scenarios and natural scenes. These challenges are further exacerbated by complex traffic environments, including camera corruption under adverse weather, noise interferences, and central bias from long-tail distributions. To address these issues, we propose a robust unsupervised attention prediction method. An Uncertainty Mining Branch refines predictions by analyzing commonalities and differences across multiple pre-trained models on natural scenes, while a Knowledge Embedding Block bridges the domain gap by incorporating driving knowledge to adaptively enhance pseudo-labels. Additionally, we introduce RoboMixup, a novel data augmentation method that improves robustness against corruption through soft attention and dynamic augmentation, and mitigates central bias by integrating random cropping into Mixup as a regularizer. To systematically evaluate robustness in self-driving attention prediction, we introduce the DriverAttention-C benchmark, comprising over 100k frames across three subsets: BDD-A-C, DR(eye)VE-C, and DADA-2000-C. Our method achieves performance equivalent to or surpassing fully supervised state-of-the-art approaches on three public datasets and the proposed robustness benchmark, reducing relative corruption degradation by 58.8% and 52.8%, and improving central bias robustness by 12.4% and 11.4% in KLD and CC metrics, respectively. Code and data are available at https://github.com/zaplm/DriverAttention.

Figures (8)

• Figure 1: Illustration of the proposed unsupervised attention prediction in self-driving. Our method bypasses the need for ground truth labels from traffic datasets, by leveraging pseudo-labels generated from models pre-trained on natural scenes. These pseudo-labels are refined through the knowledge embedding and, together with the images, are processed by the RoboMixup to address corruption and central bias. Finally, the uncertainty map and loss guide the model in learning attention regions.
• Figure 2: Overview of our proposed model. We utilize pseudo-labels from models pre-trained on natural scene datasets for unsupervised training in our approach. The Knowledge Embedding Block (KEB) is designed to integrate additional semantic information into the self-driving scenario. Our proposed data augmentation method, RoboMixup, strengthens the model's robustness against corruption and central bias by combining soft attention or random cropping with a dynamic augmentation strategy, generating realistic and challenging samples. The Attention Prediction Block (APB), using the Mobile-ViT mehta2022mobilevit backbone, consists of five stages of image feature extraction, each feeding its output to the decoder. Features from stages 1, 2, and 4 are directed to three Uncertainty blocks for multi-scale feature fusion. The Uncertainty Mining Block (UMB) uses multi-scale feature fusion and mining across multiple pseudo-labels to create an uncertainty map for each and then optimize with the uncertainty loss.
• Figure 3: Illustration of the knowledge embedding strategy: a) the process of knowledge embedding for a single pseudo-label, where the salient region can be enhanced by adding the self-driving-related instance (e.g. pedestrian) where the operator $\otimes$ means the operation in Eq. (\ref{['eqo:prior']}); b) two other examples of knowledge embedding for bicycles and motorcycles.
• Figure 4: (a) Visual Comparison of Mixup and Our Proposed Soft Attention-based Mixup. (b) Comparison of Probability Density Distribution of the Initial and Augmented Datasets.
• Figure 5: Visualization of our generated examples, highlighting how each corruption type substantially alters the original image representation.