See Less, See Right: Bi-directional Perceptual Shaping For Multimodal Reasoning

TL;DR

BiPS introduces a training-time perceptual shaping framework for vision-language models that uses two question-conditioned views of visual content to bias reasoning toward visually grounded cues. By generating precise Evidence-Preserving and Evidence-Ablated chart views through a programmatic data pipeline, BiPS defines two KL-based objectives within a coarse-to-fine GRPO curriculum: a consistency constraint that aligns predictions on preserved evidence with the original image, and a separation constraint that pushes predictions away when key evidence is removed. This design yields substantial data-efficient gains across chart understanding and general VQA benchmarks, achieving up to +8.2% average improvement with modest chart data (13K) plus additional math data (39K). Importantly, BiPS incurs no inference-time overhead and demonstrates strong out-of-domain generalization, highlighting a practical approach to robust visual grounding without relying on inference-time tools. The work suggests a promising direction for training-time perceptual shaping to enhance cross-domain multimodal reasoning in resource-constrained settings.

Abstract

Large vision-language models (VLMs) often benefit from intermediate visual cues, either injected via external tools or generated as latent visual tokens during reasoning, but these mechanisms still overlook fine-grained visual evidence (e.g., polylines in charts), generalize poorly across domains, and incur high inference-time cost. In this paper, we propose Bi-directional Perceptual Shaping (BiPS), which transforms question-conditioned masked views into bidirectional where-to-look signals that shape perception during training. BiPS first applies a KL-consistency constraint between the original image and an evidence-preserving view that keeps only question-relevant regions, encouraging coarse but complete coverage of supporting pixels. It then applies a KL-separation constraint between the original and an evidence-ablated view where critical pixels are masked so the image no longer supports the original answer, discouraging text-only shortcuts (i.e., answering from text alone) and enforcing fine-grained visual reliance. Across eight benchmarks, BiPS boosts Qwen2.5-VL-7B by 8.2% on average and shows strong out-of-domain generalization to unseen datasets and image types.

Figures (6)

• Figure 1: Illustration of different paradigms. Within each colored box, the top row illustrates the training stage and the bottom row shows the inference stage. Prior approaches are limited by shape-rigid inference-time tools and domain-specific solutions that generalize poorly.
• Figure 2: Overview of the Bi-directional Perceptual Shaping (BiPS) framework. BiPS employs a two-stage training curriculum built on the GRPO framework. Stage 1 (Consistency Stage) minimizes the KL-divergence ($L_{cons}$) between the original policy ($\pi_{\theta}$) and the policy on an evidence-preserving view ($\tilde{\pi}_{\theta}$). Stage 2 (Separation Stage) maximizes the KL-divergence ($L_{sep}$) between the original policy ($\pi_{\theta}$) and the policy on an evidence-ablated view, forcing the model to ground its reasoning in visual evidence.
• Figure 3: Overview of our data generation pipeline. This pipeline programmatically edits chart source code to generate the paired Evidence-Preserving ($I_\mathrm{{pres}}$) and Evidence-Ablated ($I_\mathrm{{abl}}$) views used for bi-directional training.
• Figure 4: Accuracy on CharXiv with respect to the weighting coefficients of the consistency $\alpha$ and separation $\beta$ constraints.
• Figure 5: Case study on ChartXiv comparing Qwen2.5-VL-7B and our BiPS-Chart. BiPS yields more visually grounded answers.