Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Thinking with Constructions: A Benchmark and Policy Optimization for Visual-Text Interleaved Geometric Reasoning

Haokun Zhao, Wanshi Xu, Haidong Yuan, Songjun Cao, Long Ma, Yanghua Xiao

Abstract

Geometric reasoning inherently requires "thinking with constructions" -- the dynamic manipulation of visual aids to bridge the gap between problem conditions and solutions. However, existing Multimodal Large Language Models (MLLMs) are largely confined to passive inference with static diagrams, lacking the strategic knowledge of when and how to construct effective visual aids. To address this, we present a framework for Visual-Text Interleaved Chain-of-Thought. We first introduce GeoAux-Bench, the first benchmark comprising 4,334 geometry problems that aligns textual construction steps with ground-truth visual updates. Our pilot study reveals two critical insights: (1) interleaved visual-textual aids outperform single-modality counterparts, which cannot losslessly capture geometric synergy; and (2) valid constructions act as entropy reducers, strongly correlating with reduced reasoning perplexity. Building on these findings, we propose Action Applicability Policy Optimization (A2PO), a reinforcement learning paradigm for mastering strategic construction. A2PO employs Adaptive Reward Shaping to regulate the timing and quality of visual aids via counterfactual sampling to distinguish necessary from redundant constructions. Experiments demonstrate our approach enables MLLMs to leverage selective auxiliary constructions, yielding a 3.51% gain over strong baselines. Code and data are available on GitHub.

Thinking with Constructions: A Benchmark and Policy Optimization for Visual-Text Interleaved Geometric Reasoning

Abstract

Geometric reasoning inherently requires "thinking with constructions" -- the dynamic manipulation of visual aids to bridge the gap between problem conditions and solutions. However, existing Multimodal Large Language Models (MLLMs) are largely confined to passive inference with static diagrams, lacking the strategic knowledge of when and how to construct effective visual aids. To address this, we present a framework for Visual-Text Interleaved Chain-of-Thought. We first introduce GeoAux-Bench, the first benchmark comprising 4,334 geometry problems that aligns textual construction steps with ground-truth visual updates. Our pilot study reveals two critical insights: (1) interleaved visual-textual aids outperform single-modality counterparts, which cannot losslessly capture geometric synergy; and (2) valid constructions act as entropy reducers, strongly correlating with reduced reasoning perplexity. Building on these findings, we propose Action Applicability Policy Optimization (A2PO), a reinforcement learning paradigm for mastering strategic construction. A2PO employs Adaptive Reward Shaping to regulate the timing and quality of visual aids via counterfactual sampling to distinguish necessary from redundant constructions. Experiments demonstrate our approach enables MLLMs to leverage selective auxiliary constructions, yielding a 3.51% gain over strong baselines. Code and data are available on GitHub.
Paper Structure (41 sections, 6 equations, 12 figures, 10 tables)

This paper contains 41 sections, 6 equations, 12 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Benchmarks for Multimodal Mathematical Reasoning
  4. Visual Chain-of-Thought (VCoT)
  5. Reinforcement Learning for Reasoning
  6. The GeoAux Benchmark and Reasoning Dynamics
  7. GeoAux-Bench Construction
  8. GeoAux-Core.
  9. GeoAux-Canvas.
  10. Quality Control and Standardization.
  11. Statistics.
  12. Pilot Study: Complementarity of Auxiliary Modalities in Interleaved Geometric Reasoning
  13. Perceptual Saliency Control (Visual Enhancement).
  14. Analysis of Reasoning Dynamics.
  15. Method
  16. ...and 26 more sections

Figures (12)

  • Figure 1: A representative sample from GeoAux-Bench. The solution trajectory is structured in a visual-text interleaved format: the textual auxiliary construction step (e.g., <aux>... </aux>) is explicitly paired with a corresponding auxiliary diagram ($I_{aux}$).
  • Figure 2: Overview of GeoAux-Bench.
  • Figure 3: Visual Enhancement Protocol. Bold red lines are used to highlight auxiliary elements ($I_{\text{aux}}^{\text{red}}$).
  • Figure 4: The framework of Action Applicability Policy Optimization (A2PO). The upper panel shows standard GRPO, while the lower panel illustrates our tri-partition sampling and adaptive reward shaping mechanism.
  • Figure 5: Attentional heatmaps of Qwen2.5-VL-7B-Instruct. Warmer colors indicate higher attention levels.
  • ...and 7 more figures