Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Sample Efficient Reinforcement Learning via Large Vision Language Model Distillation

Donghoon Lee, Tung M. Luu, Younghwan Lee, Chang D. Yoo

TL;DR

The paper addresses the high sample complexity of reinforcement learning and the impracticality of deploying large vision-language models in constrained settings. It proposes LVLM2P, a distillation framework where a lightweight student RL agent learns from both environment interactions and a pretrained LVLM teacher that suggests actions from visual observations using few-shot prompts. The LVLM teacher is queried in two stages to produce soft action distributions, which are aligned with the student policy via a KL-divergence distillation term added to the standard RL objective, enabling faster learning without relying on hand-crafted state descriptors. Experiments on four MiniGrid tasks show that LVLM2P substantially improves sample efficiency across PPO and A2C baselines, with ablations highlighting the benefits of soft distributions and a small distillation weight. The approach offers a practical path toward leveraging LVLM knowledge in resource-constrained RL applications and can be integrated with multiple RL algorithms.

Abstract

Recent research highlights the potential of multimodal foundation models in tackling complex decision-making challenges. However, their large parameters make real-world deployment resource-intensive and often impractical for constrained systems. Reinforcement learning (RL) shows promise for task-specific agents but suffers from high sample complexity, limiting practical applications. To address these challenges, we introduce LVLM to Policy (LVLM2P), a novel framework that distills knowledge from large vision-language models (LVLM) into more efficient RL agents. Our approach leverages the LVLM as a teacher, providing instructional actions based on trajectories collected by the RL agent, which helps reduce less meaningful exploration in the early stages of learning, thereby significantly accelerating the agent's learning progress. Additionally, by leveraging the LVLM to suggest actions directly from visual observations, we eliminate the need for manual textual descriptors of the environment, enhancing applicability across diverse tasks. Experiments show that LVLM2P significantly enhances the sample efficiency of baseline RL algorithms.

Sample Efficient Reinforcement Learning via Large Vision Language Model Distillation

TL;DR

The paper addresses the high sample complexity of reinforcement learning and the impracticality of deploying large vision-language models in constrained settings. It proposes LVLM2P, a distillation framework where a lightweight student RL agent learns from both environment interactions and a pretrained LVLM teacher that suggests actions from visual observations using few-shot prompts. The LVLM teacher is queried in two stages to produce soft action distributions, which are aligned with the student policy via a KL-divergence distillation term added to the standard RL objective, enabling faster learning without relying on hand-crafted state descriptors. Experiments on four MiniGrid tasks show that LVLM2P substantially improves sample efficiency across PPO and A2C baselines, with ablations highlighting the benefits of soft distributions and a small distillation weight. The approach offers a practical path toward leveraging LVLM knowledge in resource-constrained RL applications and can be integrated with multiple RL algorithms.

Abstract

Recent research highlights the potential of multimodal foundation models in tackling complex decision-making challenges. However, their large parameters make real-world deployment resource-intensive and often impractical for constrained systems. Reinforcement learning (RL) shows promise for task-specific agents but suffers from high sample complexity, limiting practical applications. To address these challenges, we introduce LVLM to Policy (LVLM2P), a novel framework that distills knowledge from large vision-language models (LVLM) into more efficient RL agents. Our approach leverages the LVLM as a teacher, providing instructional actions based on trajectories collected by the RL agent, which helps reduce less meaningful exploration in the early stages of learning, thereby significantly accelerating the agent's learning progress. Additionally, by leveraging the LVLM to suggest actions directly from visual observations, we eliminate the need for manual textual descriptors of the environment, enhancing applicability across diverse tasks. Experiments show that LVLM2P significantly enhances the sample efficiency of baseline RL algorithms.
Paper Structure (9 sections, 1 equation, 4 figures, 1 algorithm)

This paper contains 9 sections, 1 equation, 4 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Problem Formulation
  4. Methodology
  5. LVLM2P Framework
  6. LVLM as Teacher Agent
  7. Training Process of Student Policy
  8. Experiments
  9. Conclusion

Figures (4)

  • Figure 1: Given trajectories sampled by the student agent, we query the LVLM teacher during the RL update. The LVLM processes the visual observation and a textual prompt to generate the probability over a set of actions. The agent is then updated by optimizing both the conventional RL objective (e.g., PPO Schulman2017ProximalPO or A2C Mnih2016AsynchronousMF) and a knowledge distillation objective. Our proposed framework leverages the strengths of LVLMs, eliminating the need for handcrafted textual descriptors of states and significantly accelerating the student RL agent’s learning process.
  • Figure 2: An example of a prompt in GoToDoor: In the first stage, the LVLM outputs key information such as the location and orientation of the agent (red triangle) and the target object (purple door). In the second stage, the LVLM is required to provide a numerical probability over possible actions.
  • Figure 3: Top: The mean success rate, along with the standard deviation range, across all four environments (the higher the better). Bottom: Number of training samples needed with respect to mean return for all four environment (the lower the better). Each method was evaluated using three random seeds.
  • Figure 4: Ablation study of LVLM2P : Comparison of hard and soft probabilities from the LVLM teacher policy (left) and the impact of hyper-parameter $\lambda$ on success rate results (right). Each method was evaluated using three random seeds.