Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Zero-Shot Distracted Driver Detection via Vision Language Models with Double Decoupling

Takamichi Miyata, Sumiko Miyata, Andrew Morris

TL;DR

This work tackles zero-shot distracted driver detection with vision-language models by identifying two key failure modes: appearance-driven entanglement in image embeddings and collapsed, poorly separable text embeddings for fine-grained prompts. It introduces a double decoupling framework consisting of Driver-specific Appearance Decoupling (DAD) and Text Embedding Orthogonalization (TEO), along with prompt engineering tailored to VLM behavior, to emphasize distraction-relevant cues. Empirical results on the SAM-DD dataset show substantial improvements over DriveCLIP in both multiclass and binary classifications, supported by embedding visualizations and ablation studies. The approach remains lightweight and edge-friendly, offering practical road-safety benefits while acknowledging ethical limitations and the need for integrating additional context in real deployments.

Abstract

Distracted driving is a major cause of traffic collisions, calling for robust and scalable detection methods. Vision-language models (VLMs) enable strong zero-shot image classification, but existing VLM-based distracted driver detectors often underperform in real-world conditions. We identify subject-specific appearance variations (e.g., clothing, age, and gender) as a key bottleneck: VLMs entangle these factors with behavior cues, leading to decisions driven by who the driver is rather than what the driver is doing. To address this, we propose a subject decoupling framework that extracts a driver appearance embedding and removes its influence from the image embedding prior to zero-shot classification, thereby emphasizing distraction-relevant evidence. We further orthogonalize text embeddings via metric projection onto Stiefel manifold to improve separability while staying close to the original semantics. Experiments demonstrate consistent gains over prior baselines, indicating the promise of our approach for practical road-safety applications.

Zero-Shot Distracted Driver Detection via Vision Language Models with Double Decoupling

TL;DR

This work tackles zero-shot distracted driver detection with vision-language models by identifying two key failure modes: appearance-driven entanglement in image embeddings and collapsed, poorly separable text embeddings for fine-grained prompts. It introduces a double decoupling framework consisting of Driver-specific Appearance Decoupling (DAD) and Text Embedding Orthogonalization (TEO), along with prompt engineering tailored to VLM behavior, to emphasize distraction-relevant cues. Empirical results on the SAM-DD dataset show substantial improvements over DriveCLIP in both multiclass and binary classifications, supported by embedding visualizations and ablation studies. The approach remains lightweight and edge-friendly, offering practical road-safety benefits while acknowledging ethical limitations and the need for integrating additional context in real deployments.

Abstract

Distracted driving is a major cause of traffic collisions, calling for robust and scalable detection methods. Vision-language models (VLMs) enable strong zero-shot image classification, but existing VLM-based distracted driver detectors often underperform in real-world conditions. We identify subject-specific appearance variations (e.g., clothing, age, and gender) as a key bottleneck: VLMs entangle these factors with behavior cues, leading to decisions driven by who the driver is rather than what the driver is doing. To address this, we propose a subject decoupling framework that extracts a driver appearance embedding and removes its influence from the image embedding prior to zero-shot classification, thereby emphasizing distraction-relevant evidence. We further orthogonalize text embeddings via metric projection onto Stiefel manifold to improve separability while staying close to the original semantics. Experiments demonstrate consistent gains over prior baselines, indicating the promise of our approach for practical road-safety applications.
Paper Structure (19 sections, 5 equations, 2 figures, 4 tables)

This paper contains 19 sections, 5 equations, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Vision-Language Models
  4. VLMs for Distracted Driver Detection
  5. Improving Zero Shot Capability of VLMs
  6. Method
  7. Driver-specific Appearance Decoupling (DAD)
  8. Text Embedding Orthogonalization (TEO)
  9. Prompt Engineering for DDD based on VLM characteristics
  10. Experiment and Results
  11. Main Results
  12. Performance on Binary Classification
  13. Visualization of the Embedding Vectors
  14. Ablation Studies
  15. Discussion
  16. ...and 4 more sections

Figures (2)

  • Figure 1: t-SNE visualization of joint embedding spaces on SAM-DD. We randomly sample 30% of the test images and embed them with the CLIP image encoder, and we also embed all class prompts with the CLIP text encoder. Colors and marker shapes indicate classes, and hollow markers indicate text embeddings while filled markers indicate image embeddings. (a) Naive zero-shot exhibits strong clustering by driver appearance rather than by behavior class, and the text embeddings collapse into a narrow region. (b) With DAD and TEO, appearance-driven dispersion is reduced and class-wise clusters become more pronounced, while text embeddings spread out and move closer to the corresponding image clusters.
  • Figure 2: Overview of our proposed double decoupling framework for zero-shot distracted driver detection. Given an input image captured by an in-cabin camera.