Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Optimizing Visual Question Answering Models for Driving: Bridging the Gap Between Human and Machine Attention Patterns

Kaavya Rekanar, Martin Hayes, Ganesh Sistu, Ciaran Eising

TL;DR

This work addresses the gap between human and machine attention in driving-related Visual Question Answering (VQA). It introduces a human-guided feature filter that pre-selects driving-relevant visual cues before the vision transformer, and evaluates its impact using a case study on LXMERT with the nuImages dataset. The results, assessed via a subjective scoring framework, show improved alignment with human answers (lower MAE/RMSE and higher Pearson correlation) and more focused attention on driving-relevant elements, though some errors remain due to the lack of driving-specific training. The approach promises improved interpretability and robustness for driving-context VQA and motivates future fine-tuning on driving datasets and enhancement with camera-aware spatial reasoning tools.

Abstract

Visual Question Answering (VQA) models play a critical role in enhancing the perception capabilities of autonomous driving systems by allowing vehicles to analyze visual inputs alongside textual queries, fostering natural interaction and trust between the vehicle and its occupants or other road users. This study investigates the attention patterns of humans compared to a VQA model when answering driving-related questions, revealing disparities in the objects observed. We propose an approach integrating filters to optimize the model's attention mechanisms, prioritizing relevant objects and improving accuracy. Utilizing the LXMERT model for a case study, we compare attention patterns of the pre-trained and Filter Integrated models, alongside human answers using images from the NuImages dataset, gaining insights into feature prioritization. We evaluated the models using a Subjective scoring framework which shows that the integration of the feature encoder filter has enhanced the performance of the VQA model by refining its attention mechanisms.

Optimizing Visual Question Answering Models for Driving: Bridging the Gap Between Human and Machine Attention Patterns

TL;DR

This work addresses the gap between human and machine attention in driving-related Visual Question Answering (VQA). It introduces a human-guided feature filter that pre-selects driving-relevant visual cues before the vision transformer, and evaluates its impact using a case study on LXMERT with the nuImages dataset. The results, assessed via a subjective scoring framework, show improved alignment with human answers (lower MAE/RMSE and higher Pearson correlation) and more focused attention on driving-relevant elements, though some errors remain due to the lack of driving-specific training. The approach promises improved interpretability and robustness for driving-context VQA and motivates future fine-tuning on driving datasets and enhancement with camera-aware spatial reasoning tools.

Abstract

Visual Question Answering (VQA) models play a critical role in enhancing the perception capabilities of autonomous driving systems by allowing vehicles to analyze visual inputs alongside textual queries, fostering natural interaction and trust between the vehicle and its occupants or other road users. This study investigates the attention patterns of humans compared to a VQA model when answering driving-related questions, revealing disparities in the objects observed. We propose an approach integrating filters to optimize the model's attention mechanisms, prioritizing relevant objects and improving accuracy. Utilizing the LXMERT model for a case study, we compare attention patterns of the pre-trained and Filter Integrated models, alongside human answers using images from the NuImages dataset, gaining insights into feature prioritization. We evaluated the models using a Subjective scoring framework which shows that the integration of the feature encoder filter has enhanced the performance of the VQA model by refining its attention mechanisms.
Paper Structure (14 sections, 10 figures, 2 tables, 1 algorithm)

This paper contains 14 sections, 10 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background Study
  3. Proposed Methodology
  4. Object Perception and Cognitive Processes
  5. Human Answer Explanation Patterns
  6. Attention Patterns of VQA models
  7. Human-Guided Feature Filter
  8. Filter: Algorithm and Need
  9. Algorithm
  10. Case Study
  11. Dataset
  12. VQA Model: LXMERT
  13. Results and Discussion
  14. Conclusion and Future Work

Figures (10)

  • Figure 1: A demo of how VQA models work in a driving scenario
  • Figure 2: Refining VQA architecture: Integration of the filter into a general VQA architecture
  • Figure 3: Visualizing the Functionality of a LXMERT with the filter integrated: An Illustrative Approach
  • Figure 4: An Illustration of the architecture of LXMERT: self-attention with co-attention encoder
  • Figure 5: Assessment of LXMERT using the Subjective Scoring Framework
  • ...and 5 more figures