XFormParser: A Simple and Effective Multimodal Multilingual Semi-structured Form Parser

TL;DR

This work tackles Key Information Extraction from semi-structured forms in Document AI by introducing XFormParser, a simple yet effective multimodal and multilingual parser. It jointly models Semantic Entity Recognition and Relation Extraction within a Transformer-based architecture grounded in LayoutXLM, enriched with 2D positional and image embeddings, and augmented by a Bi-LSTM-based RE decoder and Biaffine scoring. A novel industrial dataset, InDFormSFT, supplements public benchmarks to better reflect real-world form parsing needs, enabling robust multilingual fine-tuning and zero-shot transfer. Empirical results show state-of-the-art or competitive performance across SER and RE in language-specific, multi-language, and zero-shot settings, demonstrating strong cross-lingual transfer and practical deployment potential; the work also provides thorough ablations and visualizations to support component necessity. GitHub resources for code, datasets, and models are publicly available for reproducibility and community use.

Abstract

In the domain of Document AI, parsing semi-structured image form is a crucial Key Information Extraction (KIE) task. The advent of pre-trained multimodal models significantly empowers Document AI frameworks to extract key information from form documents in different formats such as PDF, Word, and images. Nonetheless, form parsing is still encumbered by notable challenges like subpar capabilities in multilingual parsing and diminished recall in industrial contexts in rich text and rich visuals. In this work, we introduce a simple but effective \textbf{M}ultimodal and \textbf{M}ultilingual semi-structured \textbf{FORM} \textbf{PARSER} (\textbf{XFormParser}), which anchored on a comprehensive Transformer-based pre-trained language model and innovatively amalgamates semantic entity recognition (SER) and relation extraction (RE) into a unified framework. Combined with Bi-LSTM, the performance of multilingual parsing is significantly improved. Furthermore, we develop InDFormSFT, a pioneering supervised fine-tuning (SFT) industrial dataset that specifically addresses the parsing needs of forms in various industrial contexts. XFormParser has demonstrated its unparalleled effectiveness and robustness through rigorous testing on established benchmarks. Compared to existing state-of-the-art (SOTA) models, XFormParser notably achieves up to 1.79\% F1 score improvement on RE tasks in language-specific settings. It also exhibits exceptional cross-task performance improvements in multilingual and zero-shot settings. The codes, datasets, and pre-trained models are publicly available at https://github.com/zhbuaa0/xformparser.

Figures (8)

• Figure 1: An illustration of named entity recognition for unstructured forms.
• Figure 2: Overall architecture of the proposed XFormParser. For Multimodal input, XFormParser utilizes layoutXLM to generate vectors containing text, visual, and spatial positional information. Subsequently, these vectors are fed into the downstream joint network to complete SER and RE tasks. The SER Task obtains text box classification through fully connected layers, and the RE task learns the categories of entity relations through a decoder based on Bi-LSTM and Biaffine.
• Figure 3: Illustration of (a) The form image that is entered into the system; (b) Visualization of SER and RE results.
• Figure 4: It shows the process of data search. On the basis of the constructed data, the title of the document is extracted as the search term, and other form files are searched in the document search engine.
• Figure 5: Firstly, the optical character recognition tool is used to process the file, and the text information and border information are obtained. The data structure of the form is constructed through the border and text, including the structure information and the text information of the form.