VLM-Guided Experience Replay

TL;DR

This work introduces VLM-RB, a plug-and-play replay-prioritization layer that uses a frozen Vision–Language Model to semantically score sub-trajectories and bias experience replay in off-policy reinforcement learning. By asynchronously scoring 32-frame clips with a general prompt and mixing VLM-driven priorities with uniform replay, VLM-RB achieves substantial gains in sample efficiency and final performance across discrete and continuous tasks, while incurring only modest inference overhead. The method outperforms standard baselines like Uniform Replay and PER, particularly in long-horizon, sparse-reward settings, and ablation studies show the importance of a mixed sampling strategy and the sufficiency of 1B-scale VLMs. The results suggest that semantic priors from pre-trained VLMs can meaningfully accelerate RL learning without fine-tuning, enabling more data-efficient control in robotics and game-playing environments, with clear avenues for extension to goal-conditioned tasks and curriculum-informed prompts.

Abstract

Recent advances in Large Language Models (LLMs) and Vision-Language Models (VLMs) have enabled powerful semantic and multimodal reasoning capabilities, creating new opportunities to enhance sample efficiency, high-level planning, and interpretability in reinforcement learning (RL). While prior work has integrated LLMs and VLMs into various components of RL, the replay buffer, a core component for storing and reusing experiences, remains unexplored. We propose addressing this gap by leveraging VLMs to guide the prioritization of experiences in the replay buffer. Our key idea is to use a frozen, pre-trained VLM (requiring no fine-tuning) as an automated evaluator to identify and prioritize promising sub-trajectories from the agent's experiences. Across scenarios, including game-playing and robotics, spanning both discrete and continuous domains, agents trained with our proposed prioritization method achieve 11-52% higher average success rates and improve sample efficiency by 19-45% compared to previous approaches. https://esharony.me/projects/vlm-rb/

Figures (9)

• Figure 1: System Diagram: (1) Data is collected with the current policy $\pi_k$ interacting with multiple instances of the environment. (2) Transitions $(s,a,r,s')$ are kept in a prioritized replay buffer with a default priority $\mathbf{\bar{p}}$. (3) Asynchronously, after each insertion, a VLM worker scores the corresponding rendered clip $\tau^O$ under a prompt $\mathsf{P}$ and writes the resulting priority $\bm{\mathrm{p}^{\mathrm{VLM}}}$ back to the replay buffer (see Section \ref{['sub-sec:mechanism']}). (4) The learner samples a minibatch using the mixture distribution $q_t$ which interpolates between uniform and VLM-prioritized replay (see Section \ref{['sub-sec:mechanism']}). (5) The agent policy is updated to $\pi_{k+1}$.
• Figure 2: Temporal context resolves visual ambiguity. From the initial state $o_i$ (left), multiple futures are possible. The top sub-trajectory shows a successful grasp, while the bottom shows stagnation. By scoring sub-trajectories rather than single frames, the VLM has sufficient context to distinguish meaningful progress from failure modes.
• Figure 3: Frozen VLM scoring anticipates learned value. We visualize a single reference episode in MiniGrid/DoorKey-16x16 which contains both goal-directed actions and random noise. Left: Visualizations of two 32-frame sub-trajectories. clip A: captures a semantically relevant sequence. clip B: shows a mostly random walk which eventually reaches the goal. Right Top: A timeline of ground-truth events; sparse reward is only received at the final goal. Right Bottom: The rose curve shows the frozen VLM score for $L=32$-frame clips. The gray curves show the temporal value difference, $\Delta Q(t; L) = Q(t+\tfrac{L}{2}) - Q(t-\tfrac{L}{2})$, calculated from critic checkpoints at increasing training steps (light to dark). Early in training (light gray), the critic is uninformative, while later checkpoints (dark gray) increasingly assign positive value to the same semantic events the VLM identified. This demonstrates that the VLM provides a helpful signal long before the critic successfully converges.
• Figure 4: Success depends on alignment with semantic priors. We compare the performance of VLM-RB across frames rendered with Standard visuals (dark circles), Misleading swapped sprites (medium squares), and Abstract textures (light diamonds). The dashed line indicates the ASR of PER. Crucially, the agent's actual observations and the underlying MDP are identical across all settings; only the visual input to the VLM is altered (see Appendix \ref{['sec:vlm_prior']} for visual samples of these modifications). The significant performance drop in the Misleading and Abstract settings confirms that our method relies on the VLM correctly recognizing specific semantic objects (e.g., keys and doors) to accelerate learning.
• Figure 5: VLM-RBovercomes the failure modes of heuristic prioritization in sparse-reward tasks. Aggregated success rates on MiniGrid/DoorKey across grid sizes (8x8, 12x12, 16x16). While alternative methods (AER, ERO, ReLo) fail as they depend on dense rewards or local similarity metrics, VLM-RB successfully bridges the long-horizon dependencies. Curves show the mean across 5 seeds, with shaded standard errors.