Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LiLT: A Simple yet Effective Language-Independent Layout Transformer for Structured Document Understanding

Jiapeng Wang, Lianwen Jin, Kai Ding

TL;DR

LiLT tackles multilingual structured document understanding by decoupling text and layout and pre-training a language-independent layout Transformer. A dual-stream architecture with BiACM enables cross-modal interaction, supported by three pre-training tasks (MVLM, KPL, CAI) and a targeted optimization strategy. Pre-training on monolingual IIT-CDIP with English/RoBERTa and multilingual backbones enables strong zero-shot transfer and competitive language-specific results across eight languages and multiple SDU benchmarks. The approach reduces multilingual data collection burdens while delivering robust cross-lingual performance, and its open-source release aims to accelerate multilingual document understanding.

Abstract

Structured document understanding has attracted considerable attention and made significant progress recently, owing to its crucial role in intelligent document processing. However, most existing related models can only deal with the document data of specific language(s) (typically English) included in the pre-training collection, which is extremely limited. To address this issue, we propose a simple yet effective Language-independent Layout Transformer (LiLT) for structured document understanding. LiLT can be pre-trained on the structured documents of a single language and then directly fine-tuned on other languages with the corresponding off-the-shelf monolingual/multilingual pre-trained textual models. Experimental results on eight languages have shown that LiLT can achieve competitive or even superior performance on diverse widely-used downstream benchmarks, which enables language-independent benefit from the pre-training of document layout structure. Code and model are publicly available at https://github.com/jpWang/LiLT.

LiLT: A Simple yet Effective Language-Independent Layout Transformer for Structured Document Understanding

TL;DR

LiLT tackles multilingual structured document understanding by decoupling text and layout and pre-training a language-independent layout Transformer. A dual-stream architecture with BiACM enables cross-modal interaction, supported by three pre-training tasks (MVLM, KPL, CAI) and a targeted optimization strategy. Pre-training on monolingual IIT-CDIP with English/RoBERTa and multilingual backbones enables strong zero-shot transfer and competitive language-specific results across eight languages and multiple SDU benchmarks. The approach reduces multilingual data collection burdens while delivering robust cross-lingual performance, and its open-source release aims to accelerate multilingual document understanding.

Abstract

Structured document understanding has attracted considerable attention and made significant progress recently, owing to its crucial role in intelligent document processing. However, most existing related models can only deal with the document data of specific language(s) (typically English) included in the pre-training collection, which is extremely limited. To address this issue, we propose a simple yet effective Language-independent Layout Transformer (LiLT) for structured document understanding. LiLT can be pre-trained on the structured documents of a single language and then directly fine-tuned on other languages with the corresponding off-the-shelf monolingual/multilingual pre-trained textual models. Experimental results on eight languages have shown that LiLT can achieve competitive or even superior performance on diverse widely-used downstream benchmarks, which enables language-independent benefit from the pre-training of document layout structure. Code and model are publicly available at https://github.com/jpWang/LiLT.
Paper Structure (32 sections, 5 equations, 2 figures, 8 tables)

This paper contains 32 sections, 5 equations, 2 figures, 8 tables.

Table of Contents

  1. Introduction
  2. LiLT
  3. Model Architecture
  4. Text Embedding
  5. Layout Embedding
  6. BiACM
  7. Pre-training Tasks
  8. Masked Visual-Language Modeling
  9. Key Point Location
  10. Cross-modal Alignment Identification
  11. Optimization Strategy
  12. Experiments
  13. Pre-training Setting
  14. Ablation Study
  15. Comparisons with the SOTAs
  16. ...and 17 more sections

Figures (2)

  • Figure 1: The substitution of language does not appear obviously unnatural when the layout structure remains unchanged, as shown in a (a) form/(b) receipt. The detailed content has been re-synthesized to avoid the sensitive information leak. Best viewed in zoomed-in.
  • Figure 2: The overall illustration of our framework. Text and layout information are separately embedded and fed into the corresponding flow. BiACM is proposed to accomplish the cross-modality interaction. At the model output, text and layout features are concatenated for the self-supervised pre-training or the downstream fine-tuning. $N_l$ is the number of Transformer layers. The red *$_\mathrm{M}$/*$_\mathrm{R}$ indicates the randomly masked/replaced item for pre-training. $t$, $b$ and $r$ represent $token$, $box$ and $region$, respectively. Best viewed in zoomed-in.