Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Clarification as Supervision: Reinforcement Learning for Vision-Language Interfaces

John Gkountouras, Ivan Titov

TL;DR

AC-RL reframes vision–language interfaces as a learnable coordination problem, using clarification requests as implicit supervision to front-load the information a reasoner needs. By densifying the reward signal with a tiered scheme and freezing the clarification module, the captioner learns to produce self-sufficient initial captions that enable single-pass reasoning. Across seven mathematical visual question-answering benchmarks, AC-RL yields consistent accuracy gains (+4.4 points on average) and substantially reduces dependence on clarification during inference, especially on quantitatively rigorous tasks. The work demonstrates that interface alignment between modular AI components can be effectively learned through interaction alone, with potential for broader application beyond vision-language systems.

Abstract

Recent text-only models demonstrate remarkable mathematical reasoning capabilities. Extending these to visual domains requires vision-language models to translate images into text descriptions. However, current models, trained to produce captions for human readers, often omit the precise details that reasoning systems require. This creates an interface mismatch: reasoners often fail not due to reasoning limitations but because they lack access to critical visual information. We propose Adaptive-Clarification Reinforcement Learning (AC-RL), which teaches vision models what information reasoners need through interaction. Our key insight is that clarification requests during training reveal information gaps; by penalizing success that requires clarification, we create pressure for comprehensive initial captions that enable the reasoner to solve the problem in a single pass. AC-RL improves average accuracy by 4.4 points over pretrained baselines across seven visual mathematical reasoning benchmarks, and analysis shows it would cut clarification requests by up to 39% if those were allowed. By treating clarification as a form of implicit supervision, AC-RL demonstrates that vision-language interfaces can be effectively learned through interaction alone, without requiring explicit annotations.

Clarification as Supervision: Reinforcement Learning for Vision-Language Interfaces

TL;DR

AC-RL reframes vision–language interfaces as a learnable coordination problem, using clarification requests as implicit supervision to front-load the information a reasoner needs. By densifying the reward signal with a tiered scheme and freezing the clarification module, the captioner learns to produce self-sufficient initial captions that enable single-pass reasoning. Across seven mathematical visual question-answering benchmarks, AC-RL yields consistent accuracy gains (+4.4 points on average) and substantially reduces dependence on clarification during inference, especially on quantitatively rigorous tasks. The work demonstrates that interface alignment between modular AI components can be effectively learned through interaction alone, with potential for broader application beyond vision-language systems.

Abstract

Recent text-only models demonstrate remarkable mathematical reasoning capabilities. Extending these to visual domains requires vision-language models to translate images into text descriptions. However, current models, trained to produce captions for human readers, often omit the precise details that reasoning systems require. This creates an interface mismatch: reasoners often fail not due to reasoning limitations but because they lack access to critical visual information. We propose Adaptive-Clarification Reinforcement Learning (AC-RL), which teaches vision models what information reasoners need through interaction. Our key insight is that clarification requests during training reveal information gaps; by penalizing success that requires clarification, we create pressure for comprehensive initial captions that enable the reasoner to solve the problem in a single pass. AC-RL improves average accuracy by 4.4 points over pretrained baselines across seven visual mathematical reasoning benchmarks, and analysis shows it would cut clarification requests by up to 39% if those were allowed. By treating clarification as a form of implicit supervision, AC-RL demonstrates that vision-language interfaces can be effectively learned through interaction alone, without requiring explicit annotations.

Paper Structure

This paper contains 39 sections, 3 theorems, 16 equations, 4 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Reinforcement learning for reasoning in language models.
  4. Decoupled perception–reasoning and interface design
  5. Learning alignment through interaction.
  6. Methodology
  7. The Vision-Reasoner Interface Problem
  8. Training and Inference Protocols
  9. Clarification-Aware Reward Design
  10. Policy Optimization
  11. Experiments
  12. System Architecture.
  13. Training Configurations and Method Baselines
  14. Training Protocol.
  15. Evaluation Protocol.
  16. ...and 24 more sections

Key Result

Theorem 1

Let $\xi \sim p(\cdot \mid \tau)$ denote all post-action randomness after the policy chooses its actions (e.g., the reasoner’s sampling noise and, when clarification is used, the frozen clarification-caption sampling). Define the extended trajectory $\tilde{\tau} = (\tau, \xi)$ with joint density: where $p(\xi \mid \tau)$ is independent of $\theta$. Let the tiered reward function be defined as:

Figures (4)

  • Figure 1: Adaptive-Clarification Reinforcement Learning (AC-RL) training framework. Given an image and a question, a trainable captioner generates an initial description. During training, the frozen reasoner evaluates whether this description contains sufficient detail to solve the problem. If yes (Direct Answer Path), it attempts to answer directly, receiving reward $R=1$ for correct answers or $R=0$ for incorrect ones. If the description lacks crucial information (Clarification Path), the reasoner requests specific details, which are provided by a frozen reference captioner. Correct answers after clarification receive partial reward $R=0.7$, while incorrect answers receive $R=0$. Gradients (dotted arrows) flow only through the initial caption generation, not through clarification responses. At inference, only the direct answer path is used: the model has learned to generate sufficiently detailed initial captions, eliminating the need for clarification.
  • Figure 2: Subject-level performance comparing AC-RL to the pretrained baseline using Qwen-3B + Reasoner. Left: MathVision subjects. Right: DynaMath categories. AC-RL shows targeted improvements in quantitatively-intensive domains like geometry and algebra.
  • Figure 3: DynaMath average accuracy across education levels. AC-RL consistently outperforms the baseline regardless of problem difficulty.
  • Figure 4: Fraction of uniform-reward batches during training. AC-RL (red) maintains lower values than standard RL (blue), indicating more diverse caption generation throughout training. Both methods show increasing trends as policies converge, but AC-RL's tiered reward structure preserves more exploration.

Theorems & Definitions (5)

  • Theorem 1: Unbiasedness of the Three-Tier Reward with Stochastic Reasoner
  • proof
  • Remark 2: If the reasoner is $\theta$-dependent
  • Proposition 3: Unbiased gradient with $\theta$-dependent reasoner
  • Corollary 4