Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ThinkDrive: Chain-of-Thought Guided Progressive Reinforcement Learning Fine-Tuning for Autonomous Driving

Chang Zhao, Zheming Yang, Yunqing Hu, Qi Guo, Zijian Wang, Pengcheng Li, Wen Ji

TL;DR

ThinkDrive tackles unstructured reasoning and misalignment in autonomous driving by coupling Chain-of-Thought prompted supervised fine-tuning with a progressive reinforcement learning regimen. It introduces Gaussian-based curriculum sampling and a difficulty-aware adaptive policy optimization that modulates learning intensity based on sample difficulty, quantified via entropy across rollouts. Empirical results on DrivingVQA show ThinkDrive outperforming strong RL baselines and enabling a 2B-parameter model to surpass GPT-4o on exam metrics, highlighting improved stability, generalization, and reasoning depth in domain-specific driving tasks. This framework offers a scalable path to robust, interpretable decision-making in multimodal autonomous driving systems.

Abstract

With the rapid advancement of large language models (LLMs) technologies, their application in the domain of autonomous driving has become increasingly widespread. However, existing methods suffer from unstructured reasoning, poor generalization, and misalignment with human driving intent. While Chain-of-Thought (CoT) reasoning enhances decision transparency, conventional supervised fine-tuning (SFT) fails to fully exploit its potential, and reinforcement learning (RL) approaches face instability and suboptimal reasoning depth. We propose ThinkDrive, a CoT guided progressive RL fine-tuning framework for autonomous driving that synergizes explicit reasoning with difficulty-aware adaptive policy optimization. Our method employs a two-stage training strategy. First, we perform SFT using CoT explanations. Then, we apply progressive RL with a difficulty-aware adaptive policy optimizer that dynamically adjusts learning intensity based on sample complexity. We evaluate our approach on a public dataset. The results show that ThinkDrive outperforms strong RL baselines by 1.45%, 1.95%, and 1.01% on exam, easy-exam, and accuracy, respectively. Moreover, a 2B-parameter model trained with our method surpasses the much larger GPT-4o by 3.28% on the exam metric.

ThinkDrive: Chain-of-Thought Guided Progressive Reinforcement Learning Fine-Tuning for Autonomous Driving

TL;DR

ThinkDrive tackles unstructured reasoning and misalignment in autonomous driving by coupling Chain-of-Thought prompted supervised fine-tuning with a progressive reinforcement learning regimen. It introduces Gaussian-based curriculum sampling and a difficulty-aware adaptive policy optimization that modulates learning intensity based on sample difficulty, quantified via entropy across rollouts. Empirical results on DrivingVQA show ThinkDrive outperforming strong RL baselines and enabling a 2B-parameter model to surpass GPT-4o on exam metrics, highlighting improved stability, generalization, and reasoning depth in domain-specific driving tasks. This framework offers a scalable path to robust, interpretable decision-making in multimodal autonomous driving systems.

Abstract

With the rapid advancement of large language models (LLMs) technologies, their application in the domain of autonomous driving has become increasingly widespread. However, existing methods suffer from unstructured reasoning, poor generalization, and misalignment with human driving intent. While Chain-of-Thought (CoT) reasoning enhances decision transparency, conventional supervised fine-tuning (SFT) fails to fully exploit its potential, and reinforcement learning (RL) approaches face instability and suboptimal reasoning depth. We propose ThinkDrive, a CoT guided progressive RL fine-tuning framework for autonomous driving that synergizes explicit reasoning with difficulty-aware adaptive policy optimization. Our method employs a two-stage training strategy. First, we perform SFT using CoT explanations. Then, we apply progressive RL with a difficulty-aware adaptive policy optimizer that dynamically adjusts learning intensity based on sample complexity. We evaluate our approach on a public dataset. The results show that ThinkDrive outperforms strong RL baselines by 1.45%, 1.95%, and 1.01% on exam, easy-exam, and accuracy, respectively. Moreover, a 2B-parameter model trained with our method surpasses the much larger GPT-4o by 3.28% on the exam metric.
Paper Structure (12 sections, 8 equations, 5 figures, 2 tables)

This paper contains 12 sections, 8 equations, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. The Proposed ThinkDrive Framework
  4. Progressive Reinforcement Learning
  5. Difficulty-Aware Adaptive Policy Optimization
  6. Experiments
  7. Experimental Setup
  8. Evaluation Results
  9. Convergence Analysis
  10. End-to-End Performance
  11. Ablation Studies
  12. Conclusion

Figures (5)

  • Figure 1: Example of CoT reasoning applied in autonomous driving. The model first extracts entities from the image that are relevant to the question, then performs reasoning over these entities to derive the final answer.
  • Figure 2: Overview of the ThinkDrive framework for autonomous driving. The input consists of a driving scene image, the current query to be addressed, and task instructions. The model's reasoning capability is enhanced through a two-stage training strategy, where the most critical component is the progressive RL approach consisting of Gaussian-based curriculum sampling and the difficulty-aware adaptive policy optimization mechanism.
  • Figure 3: Training dynamics of key metrics with ThinkDrive. (a) Reward score (smoothed using a rolling average over 50 steps to reduce noise and emphasize the overall trend); (b) accuracy, exam, and easy-exam scores.
  • Figure 4: Variation of data difficulty and easy-exam metric with training steps. (a) Proportion of samples from each difficulty level at different training steps. (b) Comparison of easy-exam metric across different methods during training.
  • Figure 5: Comparison of exam metrics for open-source models.