Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient Document Parsing via Parallel Token Prediction

Lei Li, Ze Zhao, Meng Li, Zhongwang Lun, Yi Yuan, Xingjing Lu, Zheng Wei, Jiang Bian, Zang Li

Abstract

Document parsing, as a fundamental yet crucial vision task, is being revolutionized by vision-language models (VLMs). However, the autoregressive (AR) decoding inherent to VLMs creates a significant bottleneck, severely limiting parsing speed. In this paper, we propose Parallel-Token Prediction (PTP), a plugable, model-agnostic and simple-yet-effective method that enables VLMs to generate multiple future tokens in parallel with improved sample efficiency. Specifically, we insert some learnable tokens into the input sequence and design corresponding training objectives to equip the model with parallel decoding capabilities for document parsing. Furthermore, to support effective training, we develop a comprehensive data generation pipeline that efficiently produces large-scale, high-quality document parsing training data for VLMs. Extensive experiments on OmniDocBench and olmOCR-bench demonstrate that our method not only significantly improves decoding speed (1.6x-2.2x) but also reduces model hallucinations and exhibits strong generalization abilities.

Efficient Document Parsing via Parallel Token Prediction

Abstract

Document parsing, as a fundamental yet crucial vision task, is being revolutionized by vision-language models (VLMs). However, the autoregressive (AR) decoding inherent to VLMs creates a significant bottleneck, severely limiting parsing speed. In this paper, we propose Parallel-Token Prediction (PTP), a plugable, model-agnostic and simple-yet-effective method that enables VLMs to generate multiple future tokens in parallel with improved sample efficiency. Specifically, we insert some learnable tokens into the input sequence and design corresponding training objectives to equip the model with parallel decoding capabilities for document parsing. Furthermore, to support effective training, we develop a comprehensive data generation pipeline that efficiently produces large-scale, high-quality document parsing training data for VLMs. Extensive experiments on OmniDocBench and olmOCR-bench demonstrate that our method not only significantly improves decoding speed (1.6x-2.2x) but also reduces model hallucinations and exhibits strong generalization abilities.
Paper Structure (28 sections, 9 equations, 8 figures, 6 tables)

This paper contains 28 sections, 9 equations, 8 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Dataset Engine
  4. Data Curation
  5. Data Annotation
  6. Filtering and Statistics
  7. Method
  8. Preliminaries
  9. Next-Token Prediction
  10. Multi-Token Prediction
  11. Parallel-Token Prediction
  12. Inference and Analysis
  13. Experiments
  14. Experimental Settings
  15. Main Results
  16. ...and 13 more sections

Figures (8)

  • Figure 1: An illustration of the data creation pipeline.
  • Figure 2: Overview of our method. The upper illustrates the training architecture of our parallel-token prediction method, along with a visualization of the attention mask used during training. The lower depicts the inference process, where accelerated inference can be achieved by incorporating register tokens in each decoding step.
  • Figure 3: Performance comparison between NTP and PTP, including average TPOT, ITL, and latency under different QPS levels, as well as decoding speed and speedup ratio in synchronous mode, which are measured on OmniDocBench 16,886 images using an H20 GPU.
  • Figure 4: Left: Training trajectories of NTP, PTP, and MTP; Second: Performance of different methods on normal vs. hallucination-prone data; Third: PTP performance across different model architectures; Right: Register token extrapolation results.
  • Figure 5: Speculative Decoding
  • ...and 3 more figures