Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-Frame, Lightweight & Efficient Vision-Language Models for Question Answering in Autonomous Driving

Akshay Gopalkrishnan, Ross Greer, Mohan Trivedi

TL;DR

This work targets real-time autonomous driving QA by designing EM-VLM4AD, a lightweight, multi-frame vision-language system that uses a ViT-based six-view image embedder and a compact T5-based language backbone with LoRA quantization. The model aggregates multi-view embeddings through gated pooling attention and undergoes a two-stage training regime on the DriveLM dataset to align visual and textual representations. Results show substantial memory and FLOP reductions (up to 10x) with competitive or superior ROUGE-L and CIDEr scores compared to a DriveLM-Agent baseline, underscoring the value of multi-view fusion for AD QA. The approach offers practical implications for deploying VLMs in real-time driving systems and points to future extensions to video QA and retrieval-augmented techniques.

Abstract

Vision-Language Models (VLMs) and Multi-Modal Language models (MMLMs) have become prominent in autonomous driving research, as these models can provide interpretable textual reasoning and responses for end-to-end autonomous driving safety tasks using traffic scene images and other data modalities. However, current approaches to these systems use expensive large language model (LLM) backbones and image encoders, making such systems unsuitable for real-time autonomous driving systems where tight memory constraints exist and fast inference time is necessary. To address these previous issues, we develop EM-VLM4AD, an efficient, lightweight, multi-frame vision language model which performs Visual Question Answering for autonomous driving. In comparison to previous approaches, EM-VLM4AD requires at least 10 times less memory and floating point operations, while also achieving higher CIDEr and ROUGE-L scores than the existing baseline on the DriveLM dataset. EM-VLM4AD also exhibits the ability to extract relevant information from traffic views related to prompts and can answer questions for various autonomous driving subtasks. We release our code to train and evaluate our model at https://github.com/akshaygopalkr/EM-VLM4AD.

Multi-Frame, Lightweight & Efficient Vision-Language Models for Question Answering in Autonomous Driving

TL;DR

This work targets real-time autonomous driving QA by designing EM-VLM4AD, a lightweight, multi-frame vision-language system that uses a ViT-based six-view image embedder and a compact T5-based language backbone with LoRA quantization. The model aggregates multi-view embeddings through gated pooling attention and undergoes a two-stage training regime on the DriveLM dataset to align visual and textual representations. Results show substantial memory and FLOP reductions (up to 10x) with competitive or superior ROUGE-L and CIDEr scores compared to a DriveLM-Agent baseline, underscoring the value of multi-view fusion for AD QA. The approach offers practical implications for deploying VLMs in real-time driving systems and points to future extensions to video QA and retrieval-augmented techniques.

Abstract

Vision-Language Models (VLMs) and Multi-Modal Language models (MMLMs) have become prominent in autonomous driving research, as these models can provide interpretable textual reasoning and responses for end-to-end autonomous driving safety tasks using traffic scene images and other data modalities. However, current approaches to these systems use expensive large language model (LLM) backbones and image encoders, making such systems unsuitable for real-time autonomous driving systems where tight memory constraints exist and fast inference time is necessary. To address these previous issues, we develop EM-VLM4AD, an efficient, lightweight, multi-frame vision language model which performs Visual Question Answering for autonomous driving. In comparison to previous approaches, EM-VLM4AD requires at least 10 times less memory and floating point operations, while also achieving higher CIDEr and ROUGE-L scores than the existing baseline on the DriveLM dataset. EM-VLM4AD also exhibits the ability to extract relevant information from traffic views related to prompts and can answer questions for various autonomous driving subtasks. We release our code to train and evaluate our model at https://github.com/akshaygopalkr/EM-VLM4AD.
Paper Structure (14 sections, 2 equations, 3 figures, 2 tables)

This paper contains 14 sections, 2 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Research
  3. Vision-Language Models
  4. Multimodal LLMs for Autonomous Driving
  5. Multi-Image Vision-Language Models
  6. Methods
  7. Image Embedding Network
  8. Language Model
  9. Training Process
  10. Experiments
  11. Quantitative Results
  12. Computational Analysis
  13. Qualitative Results
  14. Conclusion

Figures (3)

  • Figure 1: The diagram our model uses to respond to multi-view image input and question prompts. The T5 LM is frozen during Stage 1 of training so the image embedding network learns to align with the T5 embeddings. The image patch encoder is frozen throughout all stages of training, and the Gated Pooling Attention and Projection Layer is trained in both stages.
  • Figure 2: Example correct answer generations from EM-VLM4AD. As shown these in these examples, our model is able to perform VQA for various autonomous driving tasks such as perception, planning, and traffic agent behavior prediction.
  • Figure 3: More example generations from EM-VLM4AD. As shown by the red QA examples, EM-VLM4AD can sometimes struggle with grammatical semantics and questions related to ego-vehicle behavior prediction, which may require video input for improved performance.