Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Cross-View Cross-Modal Unsupervised Domain Adaptation for Driver Monitoring System

Aditi Bhalla, Christian Hellert, Enkelejda Kasneci

TL;DR

The paper tackles driver monitoring under two challenging distribution shifts: camera viewpoint changes and sensor/modality differences. It introduces C^2UDA, a two-phase framework that first learns view-invariant representations via supervised contrastive learning on synchronized multi-view data, then aligns source and target modalities with an information bottleneck objective in an unsupervised setting. Using a unified spatiotemporal transformer backbone, the approach achieves substantial RGB gains (up to ~50% over cross-view baselines and ~5% over UDA baselines) on Drive&Act with Video Swin and MViT, and demonstrates cross-dataset transfer to the Driver Anomaly Detection dataset. These results indicate that joint cross-view and cross-modal adaptation can significantly improve the robustness and deployability of driver monitoring systems in real-world, multi-camera vehicle environments.

Abstract

Driver distraction remains a leading cause of road traffic accidents, contributing to thousands of fatalities annually across the globe. While deep learning-based driver activity recognition methods have shown promise in detecting such distractions, their effectiveness in real-world deployments is hindered by two critical challenges: variations in camera viewpoints (cross-view) and domain shifts such as change in sensor modality or environment. Existing methods typically address either cross-view generalization or unsupervised domain adaptation in isolation, leaving a gap in the robust and scalable deployment of models across diverse vehicle configurations. In this work, we propose a novel two-phase cross-view, cross-modal unsupervised domain adaptation framework that addresses these challenges jointly on real-time driver monitoring data. In the first phase, we learn view-invariant and action-discriminative features within a single modality using contrastive learning on multi-view data. In the second phase, we perform domain adaptation to a new modality using information bottleneck loss without requiring any labeled data from the new domain. We evaluate our approach using state-of-the art video transformers (Video Swin, MViT) and multi modal driver activity dataset called Drive&Act, demonstrating that our joint framework improves top-1 accuracy on RGB video data by almost 50% compared to a supervised contrastive learning-based cross-view method, and outperforms unsupervised domain adaptation-only methods by up to 5%, using the same video transformer backbone.

Cross-View Cross-Modal Unsupervised Domain Adaptation for Driver Monitoring System

TL;DR

The paper tackles driver monitoring under two challenging distribution shifts: camera viewpoint changes and sensor/modality differences. It introduces C^2UDA, a two-phase framework that first learns view-invariant representations via supervised contrastive learning on synchronized multi-view data, then aligns source and target modalities with an information bottleneck objective in an unsupervised setting. Using a unified spatiotemporal transformer backbone, the approach achieves substantial RGB gains (up to ~50% over cross-view baselines and ~5% over UDA baselines) on Drive&Act with Video Swin and MViT, and demonstrates cross-dataset transfer to the Driver Anomaly Detection dataset. These results indicate that joint cross-view and cross-modal adaptation can significantly improve the robustness and deployability of driver monitoring systems in real-world, multi-camera vehicle environments.

Abstract

Driver distraction remains a leading cause of road traffic accidents, contributing to thousands of fatalities annually across the globe. While deep learning-based driver activity recognition methods have shown promise in detecting such distractions, their effectiveness in real-world deployments is hindered by two critical challenges: variations in camera viewpoints (cross-view) and domain shifts such as change in sensor modality or environment. Existing methods typically address either cross-view generalization or unsupervised domain adaptation in isolation, leaving a gap in the robust and scalable deployment of models across diverse vehicle configurations. In this work, we propose a novel two-phase cross-view, cross-modal unsupervised domain adaptation framework that addresses these challenges jointly on real-time driver monitoring data. In the first phase, we learn view-invariant and action-discriminative features within a single modality using contrastive learning on multi-view data. In the second phase, we perform domain adaptation to a new modality using information bottleneck loss without requiring any labeled data from the new domain. We evaluate our approach using state-of-the art video transformers (Video Swin, MViT) and multi modal driver activity dataset called Drive&Act, demonstrating that our joint framework improves top-1 accuracy on RGB video data by almost 50% compared to a supervised contrastive learning-based cross-view method, and outperforms unsupervised domain adaptation-only methods by up to 5%, using the same video transformer backbone.

Paper Structure

This paper contains 7 sections, 6 equations, 1 figure, 2 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Proposed Method
  4. Experiments
  5. Results
  6. Discussion
  7. Conclusions

Figures (1)

  • Figure 1: Overview of our $\mathrm{C^2UDA}$ framework. (a) Phase-1: View-invariant pretraining is performed using supervised contrastive learning on synchronized multi-view video data (V1 as anchor, and V2 and V3 as positives) within the same modality (NIR). The shared video transformer backbone learns to produce class-discriminative and view-invariant features via a combined cross-entropy and contrastive loss. (b) Phase-2: The model is adapted to a new modality (RGB) using unlabeled target data. An IB loss is applied between source and target representations to promote domain alignment, while classification supervision is maintained on the source domain.