Doc-CoB: Enhancing Multi-Modal Document Understanding with Visual Chain-of-Boxes Reasoning

TL;DR

Doc-CoB introduces a human-inspired two-stage coarse-to-fine reasoning paradigm for multimodal document understanding that grounds answers in query-relevant layout boxes. By coupling a layout analyzer with visual prompting, and enforcing a Key Box Selecting stage followed by Focused Answering, the method preserves MLLM architectures while dramatically improving document QA and IE performance across seven benchmarks and multiple models. A fully automatic data-generation pipeline yields 249k training samples with intermediate supervision, and two enabling tasks further enhance grounding and reasoning, yielding substantial gains that can surpass much larger models in some cases. The approach is robust to various layout tools, improves efficiency through focused attention, and generalizes across domains, making it a practical enhancement for real-world document understanding systems.

Abstract

Multimodal large language models (MLLMs) have made significant progress in document understanding. However, the information-dense nature of document images still poses challenges, as most queries depend on only a few relevant regions, with the rest being redundant. Existing one-pass MLLMs process entire document images without considering query relevance, often failing to focus on critical regions and producing unfaithful responses. Inspired by the human coarse-to-fine reading pattern, we introduce Doc-CoB (Chain-of-Box), a simple-yet-effective mechanism that integrates human-style visual reasoning into MLLM without modifying its architecture. Our method allows the model to autonomously select the set of regions (boxes) most relevant to the query, and then focus attention on them for further understanding. We first design a fully automatic pipeline, integrating a commercial MLLM with a layout analyzer, to generate 249k training samples with intermediate visual reasoning supervision. Then we incorporate two enabling tasks that improve box identification and box-query reasoning, which together enhance document understanding. Extensive experiments on seven benchmarks with four popular models show that Doc-CoB significantly improves performance, demonstrating its effectiveness and wide applicability. All code, data, and models will be released publicly.

Figures (8)

• Figure 1: (a) Current MLLM one-pass inference: the MLLM generates an answer directly from the full document image. (b) Doc-CoB: a layout analyzer first segments the document layout and represents it as boxes. Key Box Selecting (S1) prompts the MLLM to select the key boxes most relevant to the query. A Blur Reserve Mask keeps these key boxes sharp while blurring the rest. Focused Answering (S2) then generates the final answer from the masked image.
• Figure 2: Automatic data generation for Doc-CoB. (a) Image Labeling. A layout analyzer segments each document image into a set of layout boxes. (b) Key-Box Labeling. Given a question, its answer, and the annotated image from (a), GPT-4o identifies the helpful and confusing boxes, providing a rationale for each choice. (c-1) Task 1: Box-ID Recognition. The model returns the index of a specified box. (c-2) Task 2: Box-Query Reasoning. The model explains how the specified box contributes to answering the query. (d) Doc-CoB Sample: A complete training example.
• Figure 3: Left: A successful case of InternVL2-8B-CoB. Right: A failure case of InternVL2-8B-CoB.
• Figure 4: Performance of InternVL2-8B* (blue) and InternVL2-8B-CoB (red) on DeepForm, VRDU-RF, SROIE and FUNSD datasets, plotted against the average token numbers.
• Figure 5: Cases showing intermediate and final outputs of Doc-CoB.