Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DocFusion: A Unified Framework for Document Parsing Tasks

Mingxu Chai, Ziyu Shen, Chong Zhang, Yue Zhang, Xiao Wang, Shihan Dou, Jihua Kang, Jiazheng Zhang, Qi Zhang

TL;DR

DocFusion presents a unified, generative framework for document parsing that jointly handles DLA, MER, TR, and OCR with a compact 289M-parameter model. It introduces GK-CEL to resolve the discrete-continuous optimization clash inherent in token-based generation of coordinates, and it leverages a new DocLatex-1.6M dataset along with DocLayNet reannotations to standardize tasks. The approach achieves state-of-the-art results across all four tasks, driven by multi-task collaboration and a cohesive architecture consisting of a vision encoder, text embeddings, and a Transformer decoder. The work demonstrates practical benefits for downstream systems, though it notes limitations in handwritten/alternative table formats and real-time applicability.

Abstract

Document parsing is essential for analyzing complex document structures and extracting fine-grained information, supporting numerous downstream applications. However, existing methods often require integrating multiple independent models to handle various parsing tasks, leading to high complexity and maintenance overhead. To address this, we propose DocFusion, a lightweight generative model with only 0.28B parameters. It unifies task representations and achieves collaborative training through an improved objective function. Experiments reveal and leverage the mutually beneficial interaction among recognition tasks, and integrating recognition data significantly enhances detection performance. The final results demonstrate that DocFusion achieves state-of-the-art (SOTA) performance across four key tasks.

DocFusion: A Unified Framework for Document Parsing Tasks

TL;DR

DocFusion presents a unified, generative framework for document parsing that jointly handles DLA, MER, TR, and OCR with a compact 289M-parameter model. It introduces GK-CEL to resolve the discrete-continuous optimization clash inherent in token-based generation of coordinates, and it leverages a new DocLatex-1.6M dataset along with DocLayNet reannotations to standardize tasks. The approach achieves state-of-the-art results across all four tasks, driven by multi-task collaboration and a cohesive architecture consisting of a vision encoder, text embeddings, and a Transformer decoder. The work demonstrates practical benefits for downstream systems, though it notes limitations in handwritten/alternative table formats and real-time applicability.

Abstract

Document parsing is essential for analyzing complex document structures and extracting fine-grained information, supporting numerous downstream applications. However, existing methods often require integrating multiple independent models to handle various parsing tasks, leading to high complexity and maintenance overhead. To address this, we propose DocFusion, a lightweight generative model with only 0.28B parameters. It unifies task representations and achieves collaborative training through an improved objective function. Experiments reveal and leverage the mutually beneficial interaction among recognition tasks, and integrating recognition data significantly enhances detection performance. The final results demonstrate that DocFusion achieves state-of-the-art (SOTA) performance across four key tasks.

Paper Structure

This paper contains 34 sections, 6 equations, 9 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Architecture
  5. Challenges and Motivations
  6. Gaussian-Kernel Design
  7. Experiments
  8. Training Datasets
  9. Evaluation Metrics
  10. Evaluation Metrics for Recognition
  11. Evaluation Metrics for Detection
  12. Selection of Baseline Models
  13. Implementation Details
  14. Main Results
  15. MER performance
  16. ...and 19 more sections

Figures (9)

  • Figure 1: The model comprises three key components: a visual encoder, a text embedding layer and a Transformer decoder. The image features extracted by the visual encoder and the instruction embeddings are combined and then passed to the Transformer decoder, which produces the final output sequence.
  • Figure 2: The distribution of logits for a target token after the loss has stabilized when using the Common CE Loss.
  • Figure 3: Illustration of Gaussian-Kernel Cross-Entropy Loss.
  • Figure 4: Validation loss curves under identical hyperparameter settings, where the only variation is the choice of the objective function.
  • Figure 5: The corresponding numbers were removed from the annotated data for mathematical expression detection.
  • ...and 4 more figures