VLM Q-Learning: Aligning Vision-Language Models for Interactive Decision-Making

TL;DR

This work tackles the challenge of aligning vision-language models (VLMs) to interactive decision-making tasks, where open-weight VLMs struggle with strict action syntax and long-horizon planning. It introduces an offline-to-online reinforcement learning framework called Advantage-Filtered Supervised Fine-Tuning (AFSFT) that augments a VLM with a critic head to filter suboptimal token-level actions, enabling the model to improve beyond its demonstrations while preserving a simple training pipeline. The method converts turns into sequences of token actions, uses parse_env/parse_agent to manage environment syntax, and optimizes a joint objective that combines imitation with TD-based value updates. Experiments across Gym Cards, BabyAI, and MiniWoB BrowserGym with two open-weight VLMs demonstrate that AFSFT effectively replaces standard SFT, enabling smooth offline-to-online learning and enabling competitive performance without relying on large proprietary models. The approach has practical impact for deploying accessible VLMs in real-world interactive tasks by reducing data quality requirements and allowing continual self-improvement through interaction.

Abstract

Recent research looks to harness the general knowledge and reasoning of large language models (LLMs) into agents that accomplish user-specified goals in interactive environments. Vision-language models (VLMs) extend LLMs to multi-modal data and provide agents with the visual reasoning necessary for new applications in areas such as computer automation. However, agent tasks emphasize skills where accessible open-weight VLMs lag behind their LLM equivalents. For example, VLMs are less capable of following an environment's strict output syntax requirements and are more focused on open-ended question answering. Overcoming these limitations requires supervised fine-tuning (SFT) on task-specific expert demonstrations. Our work approaches these challenges from an offline-to-online reinforcement learning (RL) perspective. RL lets us fine-tune VLMs to agent tasks while learning from the unsuccessful decisions of our own model or more capable (larger) models. We explore an off-policy RL solution that retains the stability and simplicity of the widely used SFT workflow while allowing our agent to self-improve and learn from low-quality datasets. We demonstrate this technique with two open-weight VLMs across three multi-modal agent domains.

Figures (12)

• Figure 1: VLM-Environment Interaction. We format text and image data from the environment into an input prompt for the VLM. The models's text reply is parsed into a concrete action executed by the environment to produce a reward and new inputs for the VLM.
• Figure 2: Converting Turns $\rightarrow$ Tokens. We treat each token of the agent's reply as a single action by passing consecutive turns of dialogue through the VLM and finding output representations corresponding to tokens where an action decision is made. Outputs corresponding to the prompt tokens are masked and ignored. The resulting sequence of RL input states is numbered $(0, 1, 2, 3, 4)$.
• Figure 3: VLM Actor-Critic. We visualize a standard decoder-only VLM. Our method adds a second output head (critic) to estimate the future return achieved by selecting each token in the vocabulary. The critic filters the dataset of the language modeling head (actor).
• Figure 4: Gym Cards Model Evaluations. We compare the task success and the rate of valid model output syntax of base models in two Gym Cards environments. LLM prompts replace the image with an equivalent text description.
• Figure 5: Offline-to-Online Gym Cards.(Left, Center Right) Task success rate ($[0, 1]$) compared against a reference score for a similar setup in zhai2024fine. A gray background indicates offline learning on a fixed dataset. A white background indicates that online environment interaction is enabled. (Center Left, Right) Action syntax accuracy during evaluation.