Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Transformers and Language Models in Form Understanding: A Comprehensive Review of Scanned Document Analysis

Abdelrahman Abdallah, Daniel Eberharter, Zoe Pfister, Adam Jatowt

TL;DR

This paper surveys transformer and language model driven approaches for form understanding in noisy scanned documents. It analyzes multi-modal architectures that fuse text, layout, and visuals, detailing architectures such as LayoutLM family, DocFormer, StrucTex, and LiLT, along with pretraining objectives like MVLM and text-image alignment. The review covers a broad set of benchmarks and datasets (FUNSD, SROIE, PubLayNet, RVL-CDIP, CORD, DocVQA, etc.) and summarizes comparative results and evaluation metrics such as Precision, Recall, F1, Accuracy, ANLS, and MAP. The work highlights the central role of layout information and cross-modal fusion in achieving robust form understanding and discusses practical trade-offs between model size and performance across tasks, offering guidance for selecting suitable solutions for noisy scanned documents.

Abstract

This paper presents a comprehensive survey of research works on the topic of form understanding in the context of scanned documents. We delve into recent advancements and breakthroughs in the field, highlighting the significance of language models and transformers in solving this challenging task. Our research methodology involves an in-depth analysis of popular documents and forms of understanding of trends over the last decade, enabling us to offer valuable insights into the evolution of this domain. Focusing on cutting-edge models, we showcase how transformers have propelled the field forward, revolutionizing form-understanding techniques. Our exploration includes an extensive examination of state-of-the-art language models designed to effectively tackle the complexities of noisy scanned documents. Furthermore, we present an overview of the latest and most relevant datasets, which serve as essential benchmarks for evaluating the performance of selected models. By comparing and contrasting the capabilities of these models, we aim to provide researchers and practitioners with useful guidance in choosing the most suitable solutions for their specific form understanding tasks.

Transformers and Language Models in Form Understanding: A Comprehensive Review of Scanned Document Analysis

TL;DR

This paper surveys transformer and language model driven approaches for form understanding in noisy scanned documents. It analyzes multi-modal architectures that fuse text, layout, and visuals, detailing architectures such as LayoutLM family, DocFormer, StrucTex, and LiLT, along with pretraining objectives like MVLM and text-image alignment. The review covers a broad set of benchmarks and datasets (FUNSD, SROIE, PubLayNet, RVL-CDIP, CORD, DocVQA, etc.) and summarizes comparative results and evaluation metrics such as Precision, Recall, F1, Accuracy, ANLS, and MAP. The work highlights the central role of layout information and cross-modal fusion in achieving robust form understanding and discusses practical trade-offs between model size and performance across tasks, offering guidance for selecting suitable solutions for noisy scanned documents.

Abstract

This paper presents a comprehensive survey of research works on the topic of form understanding in the context of scanned documents. We delve into recent advancements and breakthroughs in the field, highlighting the significance of language models and transformers in solving this challenging task. Our research methodology involves an in-depth analysis of popular documents and forms of understanding of trends over the last decade, enabling us to offer valuable insights into the evolution of this domain. Focusing on cutting-edge models, we showcase how transformers have propelled the field forward, revolutionizing form-understanding techniques. Our exploration includes an extensive examination of state-of-the-art language models designed to effectively tackle the complexities of noisy scanned documents. Furthermore, we present an overview of the latest and most relevant datasets, which serve as essential benchmarks for evaluating the performance of selected models. By comparing and contrasting the capabilities of these models, we aim to provide researchers and practitioners with useful guidance in choosing the most suitable solutions for their specific form understanding tasks.
Paper Structure (26 sections, 12 equations, 5 figures, 8 tables)

This paper contains 26 sections, 12 equations, 5 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Literature Collection Methodology
  3. Early Approaches in Document Understanding
  4. Multi-Modal Transformer Approaches
  5. Layout-Visual Fusion Models
  6. Graph-Based Models
  7. Multi-Modal Fusion Models
  8. Cross-Modal Interaction Models
  9. Sequence-to-Sequence Models
  10. Layout Representation and Language-Independent Models
  11. Hybrid Transformer Architectures
  12. Datasets
  13. Experiments Results
  14. Evaluation Metrics
  15. Precision, Recall, F1 Score, and Accuracy
  16. ...and 11 more sections

Figures (5)

  • Figure 1: Examples of scanned documents with different layouts Xu2020LayoutLMPO.
  • Figure 2: Number of relevant publications per year.
  • Figure 3: Token-to-Lineappalaraju2023docformerv2
  • Figure 4: Token-to-Gridappalaraju2023docformerv2
  • Figure 5: Absolute (top) vs. relative (bottom) 2D position representation hong2021bros.