Improving Visual Reasoning with Iterative Evidence Refinement

Abstract

Vision language models (VLMs) are increasingly capable of reasoning over images, but robust visual reasoning often requires re-grounding intermediate steps in the underlying visual evidence. Recent approaches typically rely on external image operations such as zooming or cropping to re-access fine-grained details during inference, which requires additional image re-encoding and can disrupt the reasoning trajectory. We argue that VLMs already provide strong internal signals for identifying and reusing visual evidence, and that these signals can be directly leveraged to support image-grounded reasoning. Motivated by this insight, we propose an end-to-end self-revisit framework, SIEVE, that trains models to re-engage image evidence through internal representations. SIEVE automatically extracts embeddings of salient image regions and injects them into the reasoning chain when additional grounding is needed, enabling later steps to condition on relevant visual cues without external tool calls or re-encoding. We use reinforcement learning to teach the model when to trigger visual revisiting and which region embeddings to retrieve and insert during the reasoning process. Experiments on multiple visual reasoning benchmarks, together with perception, reasoning, and hallucination evaluations, show that SIEVE yields consistent gains, improving performance by 8 percent on average across several benchmarks.

Figures (12)

• Figure 1: This figure compares tool-augmented methods with Sieve. The left shows tool-based reasoning, where external tools are invoked for additional visual information. The right shows Sieve, which directly retrieves and injects key region embeddings into the reasoning process.
• Figure 2: Performance with and without region embeddings.
• Figure 3: Training workflow of SIEVE. For each question, the embeddings of image patches aligned with key textual anchors are extracted and cached as visual evidence. During RL rollouts, the policy learns when to insert this evidence into the reasoning stream, with rewards computed from the final answer. Embeddings of visual evidence are periodically re-extracted using the updated model to keep the evidence aligned with the evolving policy.
• Figure 4: Examples of image regions associated with the extracted embeddings. The green box denotes the matched bounding box, and the red box is the expanded region whose embedding is fed into the model. The bottom-left corner shows a zoomed view of the red box. Minor box drift may occur due to Qwen-VL’s patch segmentation.
• Figure 5: (a) and (b) are the comparison of w/o inserting embedding, inserting random select embedding and insert embedding chosen by our method. (c) is IHR in different Layers on Qwen3-VL-4B-Instruct and Sieve-4B.