Training-Free Acceleration for Document Parsing Vision-Language Model with Hierarchical Speculative Decoding

TL;DR

A lightweight document parsing pipeline is employed as a draft model to predict batches of future tokens, while the more accurate VLM verifies these draft predictions in parallel, enabling parallel decoding of each region using the same draft-verify strategy.

Abstract

Document parsing is a fundamental task in multimodal understanding, supporting a wide range of downstream applications such as information extraction and intelligent document analysis. Benefiting from strong semantic modeling and robust generalization, VLM-based end-to-end approaches have emerged as the mainstream paradigm in recent years. However, these models often suffer from substantial inference latency, as they must auto-regressively generate long token sequences when processing long-form documents. In this work, motivated by the extremely long outputs and complex layout structures commonly found in document parsing, we propose a training-free and highly efficient acceleration method. Inspired by speculative decoding, we employ a lightweight document parsing pipeline as a draft model to predict batches of future tokens, while the more accurate VLM verifies these draft predictions in parallel. Moreover, we further exploit the layout-structured nature of documents by partitioning each page into independent regions, enabling parallel decoding of each region using the same draft-verify strategy. The final predictions are then assembled according to the natural reading order. Experimental results demonstrate the effectiveness of our approach: on the general-purpose OmniDocBench, our method provides a 2.42x lossless acceleration for the dots.ocr model, and achieves up to 4.89x acceleration on long-document parsing tasks. We will release our code to facilitate reproducibility and future research.

Figures (8)

• Figure 1: Overview of the proposed hierarchical speculative decoding framework for end-to-end document parsing. A lightweight pipeline first performs layout analysis and emits fixed region drafts. Stage 1 parses region crops in parallel via Decoupled Speculative Verification (DSV) to obtain verified outputs; Stage 2 aggregates them into a page-level draft and performs one full-page pass with DSV to finalize the parse. DSV combines windowed alignment and tree-structured verification to evaluate multiple draft alternatives in a single parallel multi-token step.
• Figure 2: Visualization of decoupled speculative verification. (a) Windowed alignment matches a reference window (bold) from the accepted prefix against multiple drafts to extract candidate continuations. (b) Prefix tree organizes candidates by merging common prefixes. (c) Tree-ancestry attention mask enables parallel verification, where tokens attend only to the prefix and their ancestors (green checkmarks indicate allowed attention).
• Figure 3: Impact on speedup of draft quality under different perturbations. Draft quality is measured by the draft’s score on OmniDocBench v1.5.
• Figure A1: Illustration of the repetition hallucination.
• Figure A2: High-speedup example on an English page.