Clarify or Answer: Reinforcement Learning for Agentic VQA with Context Under-specification

TL;DR

The paper addresses context-dependent VQA where external information is required to answer correctly. It introduces Clarify-or-Answer (CoA), a three-component agent that decides to answer or ask a single targeted clarification question before answering, and learns the clarification policy via GRPO-CR reinforcement learning. It provides ContextClarify, a dataset with 275 ambiguous VQA items and 275 contrast items, and demonstrates consistent gains in controller accuracy, clarification quality, and end-to-end VQA accuracy across three backbones and OOD datasets. The work shows that explicit, single-turn clarification can markedly improve reliability in multimodal reasoning and generalizes beyond the training distribution, offering a scalable approach to context underspecification in VQA.

Abstract

Real-world visual question answering (VQA) is often context-dependent: an image-question pair may be under-specified, such that the correct answer depends on external information that is not observable in the image. In such cases, directly answering can lead to confident but incorrect predictions. We propose CoA(Clarify-or-Answer), an ask-or-answer agent that separately models the decision to ask or answer, and what to ask if needed. CoA first determines whether clarification is necessary; if so, it asks a single focused question and then incorporates the response to produce the final answer. We introduce CONTEXTCLARIFY with a set of ambiguous VQA questions and the contrast set that is non-ambiguous. We further introduce GRPO-CR (Clarification Reasoning), a reinforcement learning approach that optimizes clarification question generation with multiple reward signals encouraging well-formed, focused, non-trivial questions that resolve ambiguity. Across three VLLMs and three datasets, CoA achieves consistent improvements at both the module and system levels, improving end-to-end VQA accuracy by an average of +15.3 points (83%) over prompting-based baselines

Figures (5)

• Figure 1: Comparison of direct inference (baseline) vs. our Clarify-or-Answer system. The baseline incorrectly assumes a default speed limit, leading to a wrong answer. Our CoA system identifies the ambiguity, asks for the speed limit, and answers correctly based on the additional context. Grey dashed bubble denotes the model's internal reasoning.
• Figure 2: System overview of Clarify-or-Answer (CoA).An ambiguity detection agent decides whether a visual question requires clarification. Context-missing queries trigger a clarification step, while context-sufficient queries are directly forwarded to the VQA answering agent, which produces the final answer.
• Figure 3: Training pipeline for our method GRPO-CR. The model generates clarification questions from image-query pairs, which are evaluated by multi-dimensional rewards and updated via GRPO.
• Figure 4: Case studies (ambiguity). Each block compares Baseline and Our Agent CoA via clarification question (CQ), clarification response (CR), and final answer (A). Notation: Q = question; GTCQ = ground truth clarification question; BL = baseline; CoA = clarifyOrAsk; CQ = clarification question; CR = clarification response; A = answer
• Figure 5: Case studies (non-ambiguity). Each block compares Baseline and Our Agent CoA via final answer (A). Notation: Q = question; GT = ground truth answer; BL = baseline; CoA = clarifyOrAsk; A = answer