DocParseNet: Advanced Semantic Segmentation and OCR Embeddings for Efficient Scanned Document Annotation

TL;DR

The paper tackles the challenge of accurate yet efficient annotation of scanned documents, where purely OCR or segmentation methods struggle to capture text-image context. It proposes DocParseNet, a multi-modal architecture that fuses a UNet-based visual encoder with an OCR-derived textual stream via a Shifted MLP and a DistilBERT-based textual encoder, integrated through a multi-head attention fusion at the bottleneck. On a corporate agreement dataset with over 4k annotated PDFs, it achieves $mIoU$ of $49.12$ (validation) and $49.78$ (test), using only $2.8$M parameters and $0.039$ TFLOPs, outperforming baselines by substantial margins. The approach demonstrates strong robustness across fields and significant memory and training-time efficiency, suggesting practical deployment for enterprise document processing. The work lays groundwork for scalable, multimodal document understanding and points to future expansion to additional domains and larger pre-trained backbones.

Abstract

Automating the annotation of scanned documents is challenging, requiring a balance between computational efficiency and accuracy. DocParseNet addresses this by combining deep learning and multi-modal learning to process both text and visual data. This model goes beyond traditional OCR and semantic segmentation, capturing the interplay between text and images to preserve contextual nuances in complex document structures. Our evaluations show that DocParseNet significantly outperforms conventional models, achieving mIoU scores of 49.12 on validation and 49.78 on the test set. This reflects a 58% accuracy improvement over state-of-the-art baseline models and an 18% gain compared to the UNext baseline. Remarkably, DocParseNet achieves these results with only 2.8 million parameters, reducing the model size by approximately 25 times and speeding up training by 5 times compared to other models. These metrics, coupled with a computational efficiency of 0.039 TFLOPs (BS=1), highlight DocParseNet's high performance in document annotation. The model's adaptability and scalability make it well-suited for real-world corporate document processing applications. The code is available at https://github.com/ahmad-shirazi/DocParseNet

Figures (5)

• Figure 1: Diagram illustrating the DocParseNet architecture, emphasizing its four core modules: Pixel Level Module, utilizing convolutional layers for detailed visual feature extraction; MLP phase, distilling broader image features; Text Feature Module, employing OCR for textual data extraction and contextual understanding; and Fusion Module, synergistically integrating multi-modal features to enhance document parsing and prediction accuracy.
• Figure 2: Parameters and Execution time
• Figure B1: Performance metrics of DocParseNet across 1700 epochs in the 'State' category. Subfigure (a) presents the IoU scores, while (b) illustrates the loss curves.
• Figure B2: Assessment of DocParseNet compared to baseline methods across multiple datasets using IoU.
• Figure B3: Annotated samples demonstrating different perspectives