Unbiased Visual Reasoning with Controlled Visual Inputs

TL;DR

Vision-language models often leverage spurious visual correlations rather than true causal evidence in visual question answering. VISTA introduces a modular perception–reasoning framework that decouples a frozen VLM sensor from a text-only LLM reasoner via an explicit information bottleneck, restricting visual input to simple, objective perception queries and driving iterative, evidence-seeking reasoning. Trained with Group Relative Policy Optimization on a curated multi-step question set, VISTA achieves substantial robustness gains on SpuriVerse while remaining competitive on MMVP and SeedBench, and it transfers robustly to unseen sensors. Theoretically, the framework yields an interface-based generalization bound depending on the budget $C_T = T \log |\,\Sigma_{ot}\,|$, formalizing how limiting visual information reduces overfitting to spurious cues. Collectively, VISTA provides a practical, transferable approach to unbiased visual reasoning with improved credit assignment and interpretable reasoning traces.

Abstract

End-to-end Vision-language Models (VLMs) often answer visual questions by exploiting spurious correlations instead of causal visual evidence, and can become more shortcut-prone when fine-tuned. We introduce VISTA (Visual-Information Separation for Text-based Analysis), a modular framework that decouples perception from reasoning via an explicit information bottleneck. A frozen VLM sensor is restricted to short, objective perception queries, while a text-only LLM reasoner decomposes each question, plans queries, and aggregates visual facts in natural language. This controlled interface defines a reward-aligned environment for training unbiased visual reasoning with reinforcement learning. Instantiated with Qwen2.5-VL and Llama3.2-Vision sensors, and trained with GRPO from only 641 curated multi-step questions, VISTA significantly improves robustness to real-world spurious correlations on SpuriVerse (+16.29% with Qwen-2.5-VL-7B and +6.77% with Llama-3.2-Vision-11B), while remaining competitive on MMVP and a balanced SeedBench subset. VISTA transfers robustly across unseen VLM sensors and is able to recognize and recover from VLM perception failures. Human analysis further shows that VISTA's reasoning traces are more neutral, less reliant on spurious attributes, and more explicitly grounded in visual evidence than end-to-end VLM baselines.

Figures (4)

• Figure 1: Comparison between an end-to-end VLM and VISTA on a SpuriVerse example (actual model outputs). Spurious attributes are highlighted in red. Bottom: The end-to-end Qwen2.5-VL model predicts Yes by exploiting spurious attributes (e.g., scaffolding and stereotypical attire) that are irrelevant to the question, resulting in an error. Top: VISTA decouples perception from reasoning via an information bottleneck and follows a neutral, iterative decision process: the LLM reasoner emits CoT rationales before each action, issues targeted simple visual queries as actions, and terminates the interaction once a conclusion is reached. By explicitly checking the men's actions and interactions, the reasoner remains invariant to the spurious cues and correctly predicts No.
• Figure 2: Accepted vs. rejected queries. The top row shows rejected cases, and the bottom row shows accepted cases. The vision-only sensor answers perception questions in six categories and may emit one brief Overview when the text is under-specified; all requests requiring high-level inference are Rejected.
• Figure 3: Bars show the change in percentage points of each trained model relative to its base policy on three robustness benchmarks. VISTA with RL yields consistent, sizable robustness gains, whereas end-to-end gives negligible or even negative improvements.
• Figure 4: Input image for the example where the VISTA reasoner recovers from VLM errors. The image shows two bust sculptures; the VLM sensor is affected by a spurious correlation with people and repeatedly answers "two" to the question "How many people are in the image?", even though the correct answer is zero. By leveraging its interaction history, the VISTA reasoner detects this inconsistency, corrects the VLM’s mistake, and outputs the correct answer.

Theorems & Definitions (3)

• Theorem : Informal, generalization under an information bottleneck
• Lemma A.1: Conditional MI generalization bound steinke2020reasoning
• Proposition A.2: Interface-capacity generalization bound