COVLM-RL: Critical Object-Oriented Reasoning for Autonomous Driving Using VLM-Guided Reinforcement Learning

TL;DR

The paper tackles generalization, data efficiency, and interpretability challenges in end-to-end autonomous driving by fusing a Vision-Language Model with reinforcement learning through critical object-oriented Chain-of-Thought reasoning. It introduces COVLM-RL, which derives high-level semantic priors from multi-view inputs and guides RL via a semantic-consistency loss that aligns plans with actions. The approach uses a lightweight VLM (Mini-InternVL) and multi-view encoding to produce structured decisions, improving training stability and policy interpretability. In CARLA, COVLM-RL substantially improves success rates in both seen (≈30% gain) and unseen (≈50% gain) environments, demonstrating strong generalization and robust driving behavior.

Abstract

End-to-end autonomous driving frameworks face persistent challenges in generalization, training efficiency, and interpretability. While recent methods leverage Vision-Language Models (VLMs) through supervised learning on large-scale datasets to improve reasoning, they often lack robustness in novel scenarios. Conversely, reinforcement learning (RL)-based approaches enhance adaptability but remain data-inefficient and lack transparent decision-making. % contribution To address these limitations, we propose COVLM-RL, a novel end-to-end driving framework that integrates Critical Object-oriented (CO) reasoning with VLM-guided RL. Specifically, we design a Chain-of-Thought (CoT) prompting strategy that enables the VLM to reason over critical traffic elements and generate high-level semantic decisions, effectively transforming multi-view visual inputs into structured semantic decision priors. These priors reduce the input dimensionality and inject task-relevant knowledge into the RL loop, accelerating training and improving policy interpretability. However, bridging high-level semantic guidance with continuous low-level control remains non-trivial. To this end, we introduce a consistency loss that encourages alignment between the VLM's semantic plans and the RL agent's control outputs, enhancing interpretability and training stability. Experiments conducted in the CARLA simulator demonstrate that COVLM-RL significantly improves the success rate by 30\% in trained driving environments and by 50\% in previously unseen environments, highlighting its strong generalization capability.

Figures (5)

• Figure 1: Comparison of existing End-to-End driving policies: (a)Vision-based End-to-End driving policy. (b) VLM-guided End-to-End driving policy. (c) Our proposed COVLM-RL, a novel end-to-end driving framework that integrates critical object-oriented reasoning with VLM-guided RL.
• Figure 2: Overview of the proposed COVLM-RL framework. The system integrates a critical object-oriented CoT reasoning module with a pretrained VLM to generate high-level semantic guidance from multi-view camera inputs. The CoT module sequentially identifies key traffic objects, predicts their behavior, and plans an appropriate ego action in natural language. This action is parsed into both a discrete meta-action (one-hot) and a semantic embedding. The meta-action and its associated CoT features are concatenated with environment observations (physical state and global waypoint) and used as input to the RL policy. During training, a consistency loss encourages alignment between the RL agent's control outputs and the VLM's semantic intent.
• Figure 3: Overview of the parsing process for high-level semantic decisions produced by the VLM.
• Figure 4: Training curves of different methods:(a) Episode Reward (b) Speed (c) Survived distance.
• Figure 5: Evaluation Metrics of different algorithms: (a) Trained scenarios (b) Unseen scenarios.