ReasonDrive: Efficient Visual Question Answering for Autonomous Vehicles with Reasoning-Enhanced Small Vision-Language Models

TL;DR

This work examines whether incorporating explicit reasoning during fine-tuning improves small vision-language models for autonomous driving. By generating structured reasoning chains with GPT-4o and category-specific prompts, the authors create a reasoning-enhanced dataset and fine-tune several small VLMs using a LoRA-based approach. Across model families (Llama3.2-11B, Llava1.5-7B, Qwen2.5VL), reasoning-based fine-tuning consistently improves accuracy and text-generation quality, with the largest gain observed in the Llama3.2-11B-reason model. The results highlight the value of transparent, step-by-step reasoning for safety-critical driving decisions and point to a practical path toward more interpretable, deployable autonomous driving systems.

Abstract

Vision-language models (VLMs) show promise for autonomous driving but often lack transparent reasoning capabilities that are critical for safety. We investigate whether explicitly modeling reasoning during fine-tuning enhances VLM performance on driving decision tasks. Using GPT-4o, we generate structured reasoning chains for driving scenarios from the DriveLM benchmark with category-specific prompting strategies. We compare reasoning-based fine-tuning, answer-only fine-tuning, and baseline instruction-tuned models across multiple small VLM families (Llama 3.2, Llava 1.5, and Qwen 2.5VL). Our results demonstrate that reasoning-based fine-tuning consistently outperforms alternatives, with Llama3.2-11B-reason achieving the highest performance. Models fine-tuned with reasoning show substantial improvements in accuracy and text generation quality, suggesting explicit reasoning enhances internal representations for driving decisions. These findings highlight the importance of transparent decision processes in safety-critical domains and offer a promising direction for developing more interpretable autonomous driving systems.

Figures (4)

• Figure 1: Example frame from our reasoning-enhanced dataset derived from DriveLM. Each frame consists of six camera views (top) and four categories of driving tasks (bottom) with the original question, generated reasoning, and final answer.
• Figure 2: Comparing reasoning approaches within the Llama3.2-11B model family for a parking area scenario. The reason variant correctly identifies the need to stop completely through structured reasoning about the parking environment and pedestrian safety. The simple variant incorrectly assumes no safety issues and recommends maintaining speed with minimal reasoning. The Instruct variant provides general cautionary advice without the specific recommendation to stop completely.
• Figure 3: Comparing model responses for safe driving actions in a wet road scenario. The reason models consistently recognize the need for gradual deceleration without braking on wet surfaces, while simple models vary in quality of response and Instruct models often add unnecessary actions or misinterpret the scene.
• Figure 4: Comparing model responses for safe driving actions in a nighttime urban road scenario. The reason models consistently identify proper speed maintenance and gradual deceleration, while simple and Instruct variants show inconsistency and often recommend unnecessary braking that could be unsafe on wet roads.