Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

COVR:Collaborative Optimization of VLMs and RL Agent for Visual-Based Control

Canming Xia, Peixi Peng, Guang Tan, Zhan Su, Haoran Xu, Zhenxian Liu, Luntong Li

TL;DR

COVR tackles the inefficiency of visual reinforcement learning by creating a collaborative loop between a vision-language model (VLM) and a reinforcement learning agent. The framework jointly tunes the VLM with RL-generated trajectories (RL-tuned VLM) and uses the enhanced VLM to provide action priors that guide the RL policy (VLM-guided RL). Two key modules—Exploration-Driven Dynamic Filter (EDDF) and Return-Aware Adaptive Loss Weight (RALW)—select high-quality data and stabilize fine-tuning by weighting samples by return, while progressive fine-tuning reduces resource use. Empirical results on CARLA and DMControl demonstrate state-of-the-art performance and robust ablations, highlighting improved sample efficiency, policy guidance, and cross-scenario generalization due to the bidirectional VLM-RL optimization.

Abstract

Visual reinforcement learning (RL) suffers from poor sample efficiency due to high-dimensional observations in complex tasks. While existing works have shown that vision-language models (VLMs) can assist RL, they often focus on knowledge distillation from the VLM to RL, overlooking the potential of RL-generated interaction data to enhance the VLM. To address this, we propose COVR, a collaborative optimization framework that enables the mutual enhancement of the VLM and RL policies. Specifically, COVR fine-tunes the VLM with RL-generated data to enhance the semantic reasoning ability consistent with the target task, and uses the enhanced VLM to further guide policy learning via action priors. To improve fine-tuning efficiency, we introduce two key modules: (1) an Exploration-Driven Dynamic Filter module that preserves valuable exploration samples using adaptive thresholds based on the degree of exploration, and (2) a Return-Aware Adaptive Loss Weight module that improves the stability of training by quantifying the inconsistency of sampling actions via return signals of RL. We further design a progressive fine-tuning strategy to reduce resource consumption. Extensive experiments show that COVR achieves strong performance across various challenging visual control tasks.

COVR:Collaborative Optimization of VLMs and RL Agent for Visual-Based Control

TL;DR

COVR tackles the inefficiency of visual reinforcement learning by creating a collaborative loop between a vision-language model (VLM) and a reinforcement learning agent. The framework jointly tunes the VLM with RL-generated trajectories (RL-tuned VLM) and uses the enhanced VLM to provide action priors that guide the RL policy (VLM-guided RL). Two key modules—Exploration-Driven Dynamic Filter (EDDF) and Return-Aware Adaptive Loss Weight (RALW)—select high-quality data and stabilize fine-tuning by weighting samples by return, while progressive fine-tuning reduces resource use. Empirical results on CARLA and DMControl demonstrate state-of-the-art performance and robust ablations, highlighting improved sample efficiency, policy guidance, and cross-scenario generalization due to the bidirectional VLM-RL optimization.

Abstract

Visual reinforcement learning (RL) suffers from poor sample efficiency due to high-dimensional observations in complex tasks. While existing works have shown that vision-language models (VLMs) can assist RL, they often focus on knowledge distillation from the VLM to RL, overlooking the potential of RL-generated interaction data to enhance the VLM. To address this, we propose COVR, a collaborative optimization framework that enables the mutual enhancement of the VLM and RL policies. Specifically, COVR fine-tunes the VLM with RL-generated data to enhance the semantic reasoning ability consistent with the target task, and uses the enhanced VLM to further guide policy learning via action priors. To improve fine-tuning efficiency, we introduce two key modules: (1) an Exploration-Driven Dynamic Filter module that preserves valuable exploration samples using adaptive thresholds based on the degree of exploration, and (2) a Return-Aware Adaptive Loss Weight module that improves the stability of training by quantifying the inconsistency of sampling actions via return signals of RL. We further design a progressive fine-tuning strategy to reduce resource consumption. Extensive experiments show that COVR achieves strong performance across various challenging visual control tasks.
Paper Structure (47 sections, 8 equations, 10 figures, 18 tables, 1 algorithm)

This paper contains 47 sections, 8 equations, 10 figures, 18 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. LM-assisted RL.
  4. Training LMs with RL.
  5. Preliminary
  6. Process of Visual RL.
  7. Soft Actor Critic.
  8. Method
  9. Overview
  10. Policy Guidance
  11. Exploration-Driven Dynamic Filter
  12. Return-Aware Adaptive Loss Weight
  13. Progressive Fine-tuning
  14. Experiments
  15. Experimental Settings
  16. ...and 32 more sections

Figures (10)

  • Figure 1: Comparison of VLM-adapted and VLM-assisted policy learners.
  • Figure 2: Collaborative optimization framework of COVR. It consists of two main components: (1) VLM-Guided RL. During this stage, the agent learns the policy under the guidance of actions inferred by the VLM. (2) RL-tuned VLM. This part comprises two essential modules: EDDF and RALW. Through the interaction of these two modules, the expertise of the VLM in specific domains is improved, which in turn benefits RL.
  • Figure 3: Visualization of the average return curve during training in CARLA.
  • Figure 4: Schematic illustration of VLMs' generalization capability. After fine-tuning, the model demonstrates the ability to generalize its inference performance to unseen yet structurally similar scenarios, such as both being straight road conditions.
  • Figure 5: Visualization of the CARLA scenarios.
  • ...and 5 more figures