Best-of-Q: Improving VLM agents with Q-function Action Ranking at Inference

TL;DR

Best-of-Q tackles the challenge of adapting Vision-Language Model (VLM) agents to dynamic web environments without costly retraining. It decouples perception from decision-making by freezing the VLM as an action proposer and training a lightweight Q-function offline to rerank a set of N candidate actions at inference. Using Implicit Q-Learning (IQL) on offline data, the approach delivers immediate policy improvements, demonstrated by substantial gains on the WebVoyager benchmark across GPT-4.1 and open-source backbones, with strong cost-efficiency relative to fine-tuning. The findings highlight that while the method yields meaningful improvements, the ultimate limit lies in the quality of action proposals from the base VLM, pointing to future work on enhancing proposal diversity alongside value-based selection.

Abstract

Vision-Language Models (VLMs) have become powerful backbones for agents to autonomously operate in digital environments like the web and operating systems. However, these models suffer from inadaptability to fast-changing environments like the web, which can be alleviated by fine-tuning requiring expansive model training and data collection. In this work, we introduce a novel paradigm for enhancing agentic VLM policies at inference without policy retraining. Fundamentally, our approach decouples the VLM's role as a high-capacity action proposer from the final action selection mechanism. We keep the VLM policy frozen and use it to generate a set of candidate actions for a given state. Then, a lightweight, offline-trained Q-function reranks these candidates, and the agent executes the action with the highest estimated value. The main contribution is to apply the Q-function directly during inference for immediate policy improvement, and not offline to relabel data for policy retraining. We demonstrate on the academic WebVoyager benchmark that our method significantly boosts agent success rates, improving a Qwen2.5-VL-7B agent from 38.8% to 55.7% and a proprietary GPT-4.1 agent from 82.4% to 88.8%.

Figures (12)

• Figure 1: Structure of the Best-of-Q Agent. The multi-modal state goes through a VLM policy, prompted to give $N$ action candidates. A frozen VLM processes raw state information and those action candidates into separate embeddings. These fixed-size vectors are then concatenated and fed into a lightweight Multi-Layer Perceptron (MLP) which outputs the Q-value used for reranking candidate actions at inference time.
• Figure 2: Pareto plot of the Best-of-Q Agent against prompting baselines, and VLM-action selector baselines. The cost is in USD/benchmark averaged on 3 runs. The points associated to "VLM-7B", "VLM-72B" and "VLM-GPT" are references to agents using a VLM as an action selector in place of a trained Q-function. For further insights on those agents and that study, refer to Section \ref{['sec:vlm-as-sampler']}.
• Figure 3: Breakdown of agent performance on a challenging Google Flights task. The analysis shows that in 50.2% of steps, the base VLM (Qwen2.5-VL-7B) fails by not proposing the correct "golden" action (as defined by a stronger GPT-4.1 policy). When the correct action is proposed, our Q-function successfully selects it in 13.6% of cases but fails to do so in the remaining 36.2%. This demonstrates that the primary performance bottleneck is the VLM's limited proposal ability, not the Q-function's selection process
• Figure 4: Performance of the Best-of-Q Agent as a function of the number of trajectories used for Q-function training. We see that the agents are self-improving the more data it generated and on which it is trained.
• Figure 5: The $\epsilon$-greedy agent used for data collection. A typical value of $\epsilon$ ranges between 0.3 and 0.5.