LayoutLLM: Layout Instruction Tuning with Large Language Models for Document Understanding

TL;DR

LayoutLLM tackles zero-shot document understanding by explicitly modeling document layout through a two-stage layout instruction tuning framework. It integrates a document pre-trained encoder (LayoutLMv3) with an LLM/MLLM and introduces three hierarchical pre-training tasks (document-level, region-level, segment-level) plus the LayoutCoT-based supervised fine-tuning to guide region-focused reasoning. Empirical results on multiple document VQA and VIE benchmarks show consistent improvements over open-source LLM/MLLM baselines, with additional interpretability and potential for human-in-the-loop correction. The approach demonstrates that incorporating structured layout information into instruction-tuned LLMs yields robust, layout-aware zero-shot document understanding with practical impact for document AI systems.

Abstract

Recently, leveraging large language models (LLMs) or multimodal large language models (MLLMs) for document understanding has been proven very promising. However, previous works that employ LLMs/MLLMs for document understanding have not fully explored and utilized the document layout information, which is vital for precise document understanding. In this paper, we propose LayoutLLM, an LLM/MLLM based method for document understanding. The core of LayoutLLM is a layout instruction tuning strategy, which is specially designed to enhance the comprehension and utilization of document layouts. The proposed layout instruction tuning strategy consists of two components: Layout-aware Pre-training and Layout-aware Supervised Fine-tuning. To capture the characteristics of document layout in Layout-aware Pre-training, three groups of pre-training tasks, corresponding to document-level, region-level and segment-level information, are introduced. Furthermore, a novel module called layout chain-of-thought (LayoutCoT) is devised to enable LayoutLLM to focus on regions relevant to the question and generate accurate answers. LayoutCoT is effective for boosting the performance of document understanding. Meanwhile, it brings a certain degree of interpretability, which could facilitate manual inspection and correction. Experiments on standard benchmarks show that the proposed LayoutLLM significantly outperforms existing methods that adopt open-source 7B LLMs/MLLMs for document understanding. The training data of the LayoutLLM is publicly available at https://github.com/AlibabaResearch/AdvancedLiterateMachinery/tree/main/DocumentUnderstanding/LayoutLLM

Figures (15)

• Figure 1: LLMs/MLLMs for document understanding. The LayoutLLM is an LLM/MLLM based method that integrates a document pre-trained model as encoder. It is trained by the newly proposed layout instruction tuning strategy which consists of Layout-aware Pre-training and Layout-aware Supervised Fine-tuning.
• Figure 2: Overall architecture of LayoutLLM.
• Figure 3: Overview of the Layout Instruction Tuning. (a) Document-level, region-level, and segment-level pre-training tasks, unified in instruction tuning format, are introduced. (b) A novel module called LayoutCoT is designed to enable LayoutLLM to focus on regions relevant to the question and generate accurate answers through three intermediate steps.
• Figure 4: Qualitative results on DocVQA. Green boxes are the areas concentrated in the step 2 of LayoutCoT.
• Figure 5: Interactive correction with LayoutCoT. Green represents the correct relevant areas and answers, while Red represents the original incorrect ones. Best viewed in digital version.