Small Drafts, Big Verdict: Information-Intensive Visual Reasoning via Speculation

TL;DR

The paper tackles information-intensive visual reasoning by introducing Speculative Verdict (SV), a training-free, two-stage framework that combines multiple lightweight draft VLMs with a single strong verdict model. Draft experts generate diverse reasoning paths to widen evidence coverage, while the verdict synthesizes and corrects across paths, significantly improving accuracy and robustness with favorable cost efficiency. SV achieves consistent gains on InfographicVQA, ChartMuseum, ChartQAPro, and HR-Bench 4K, and generalizes to multimodal reasoning tasks like TallyQA and MathVista, while offering strong corrections in minority-correct and some zero-correct cases. The approach demonstrates a practical, scalable paradigm for robust multimodal reasoning without extensive task-specific training.

Abstract

Large Vision-Language Models (VLMs) have achieved remarkable progress in multimodal understanding, yet they struggle when reasoning over information-intensive images that densely interleave textual annotations with fine-grained graphical elements. The main challenges lie in precisely localizing critical cues in dense layouts and multi-hop reasoning to integrate dispersed evidence. We propose Speculative Verdict (SV), a training-free framework inspired by speculative decoding that combines multiple lightweight draft experts with a large verdict model. In the draft stage, small VLMs act as draft experts to generate reasoning paths that provide diverse localization candidates; in the verdict stage, a strong VLM synthesizes these paths to produce the final answer, minimizing computational cost while recovering correct answers. To further improve efficiency and accuracy, SV introduces a consensus expert selection mechanism that forwards only high-agreement reasoning paths to the verdict. Empirically, SV achieves consistent gains on challenging information-intensive and high-resolution visual question answering benchmarks, including InfographicVQA, ChartMuseum, ChartQAPro, and HR-Bench 4K. By synthesizing correct insights from multiple partially accurate reasoning paths, SV achieves both error correction and cost-efficiency compared to large proprietary models or training pipelines. Code is available at https://github.com/Tinaliu0123/speculative-verdict.

Figures (13)

• Figure 1: Examples of correct reasoning paths for information-intensive VQA tasks. They illustrate distinct paths: (a) focuses on the localization of a specific chart, symbol identification, and reasoning from a single percentage value; (b) focuses on keyword-based localization, evidence aggregation from multiple entries across the entire image, and cross-entity sorting to select the minimum.
• Figure 2: Overview of Speculative Verdict (SV). Inspired by speculative decoding, SV operates in two stages. In the draft stage, given an input question-image pair, $k$ small candidate VLMs first generate candidate answers, from which we compute a global consensus score $s(y_i)$ for each answer based on pairwise NLL differences. We then select $m$ draft experts with the strongest consensus to generate reasoning paths. In the verdict stage, the large verdict model verifies and integrates these paths to yield the final answer.
• Figure 3: An illustration of Speculative Verdict on InfographicVQA. Five candidate VLMs first produce candidate answers, with only two providing the correct result. Consensus scoring ranks answers by agreement, and the three with the lowest scores are selected as draft experts. Although some experts commit extraction errors (confusing player’s share with NFL revenue), the verdict synthesizes their reasoning paths and successfully recovers the correct answer (49%). This illustrates SV’s ability to identify reliable experts and achieve error correction.
• Figure 4: SV’s correction ability on verdict's error cases across information-intensive benchmarks (GPT-4o as verdict). We consider only cases where the verdict itself fails, to isolate SV’s independent correction capacity. For each benchmark, three bars denote expert correctness categories (majority-correct, minority-correct, and zero-correct), defined by how many selected experts provide the correct answer. Within each category, the bars are split into the proportion corrected by SV (dark) versus not corrected (light). More details can be found in Appendix \ref{['app:recovery']}.
• Figure 5: Ablations on the number of draft experts $m$.