Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VLGOR: Visual-Language Knowledge Guided Offline Reinforcement Learning for Generalizable Agents

Pengsen Liu, Maosen Zeng, Nan Tang, Kaiyuan Li, Jing-Cheng Pang, Yunan Liu, Yang Yu

Abstract

Combining Large Language Models (LLMs) with Reinforcement Learning (RL) enables agents to interpret language instructions more effectively for task execution. However, LLMs typically lack direct perception of the physical environment, which limits their understanding of environmental dynamics and their ability to generalize to unseen tasks. To address this limitation, we propose Visual-Language Knowledge-Guided Offline Reinforcement Learning (VLGOR), a framework that integrates visual and language knowledge to generate imaginary rollouts, thereby enriching the interaction data. The core premise of VLGOR is to fine-tune a vision-language model to predict future states and actions conditioned on an initial visual observation and high-level instructions, ensuring that the generated rollouts remain temporally coherent and spatially plausible. Furthermore, we employ counterfactual prompts to produce more diverse rollouts for offline RL training, enabling the agent to acquire knowledge that facilitates following language instructions while grounding in environments based on visual cues. Experiments on robotic manipulation benchmarks demonstrate that VLGOR significantly improves performance on unseen tasks requiring novel optimal policies, achieving a success rate over 24% higher than the baseline methods.

VLGOR: Visual-Language Knowledge Guided Offline Reinforcement Learning for Generalizable Agents

Abstract

Combining Large Language Models (LLMs) with Reinforcement Learning (RL) enables agents to interpret language instructions more effectively for task execution. However, LLMs typically lack direct perception of the physical environment, which limits their understanding of environmental dynamics and their ability to generalize to unseen tasks. To address this limitation, we propose Visual-Language Knowledge-Guided Offline Reinforcement Learning (VLGOR), a framework that integrates visual and language knowledge to generate imaginary rollouts, thereby enriching the interaction data. The core premise of VLGOR is to fine-tune a vision-language model to predict future states and actions conditioned on an initial visual observation and high-level instructions, ensuring that the generated rollouts remain temporally coherent and spatially plausible. Furthermore, we employ counterfactual prompts to produce more diverse rollouts for offline RL training, enabling the agent to acquire knowledge that facilitates following language instructions while grounding in environments based on visual cues. Experiments on robotic manipulation benchmarks demonstrate that VLGOR significantly improves performance on unseen tasks requiring novel optimal policies, achieving a success rate over 24% higher than the baseline methods.
Paper Structure (23 sections, 3 equations, 8 figures, 6 tables)

This paper contains 23 sections, 3 equations, 8 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Offline Reinforcement Learning (RL)
  4. Large Language Models (LLMs) for RL
  5. Method
  6. Problem Formulation
  7. Rollout Generation via Counterfactual Prompts
  8. Skill Acquisition through Offline RL
  9. Experiment
  10. Experimental Setting
  11. Offline RL Performance Comparison
  12. Rollout Generation Quality Analysis
  13. Model Analysis
  14. Conclusion and Limitation
  15. Implementation Details.
  16. ...and 8 more sections

Figures (8)

  • Figure 1: An example comparison of imaginary rollouts generated by KALM pang2024kalm and our VLGOR. Relying solely on language knowledge, KALM does not explicitly model the spatial states of multiple objects. In contrast, VLGOR generates more realistic rollouts with smoother and physically plausible object motions.
  • Figure 2: Overall framework of the proposed VLGOR method. It consists of two stages: rollout generation for synthesizing imaginary rollouts from novel instructions, and skill acquisition for learning policies via offline reinforcement learning.
  • Figure 3: Illustration of BAGEL fine-tuning optimization via vertical data flow and rollout generation via horizontal data flow.
  • Figure 4: Success rates (%) of different methods on unseen tasks with varying difficulty levels. The top row reports success rate on the CLEVR-Robot environment, while the bottom row reports success rate on the Meta-World environment. Results are averaged over the last five checkpoints, with error bars indicating half the standard deviation across three random seeds.
  • Figure 5: Comparison of imaginary rollouts generated by KALM pang2024kalm and the proposed VLGOR.
  • ...and 3 more figures