ROAP: A Reading-Order and Attention-Prior Pipeline for Optimizing Layout Transformers in Key Information Extraction

TL;DR

This paper addresses two core challenges in multimodal document understanding: the lack of explicit reading order and attention interference from visual tokens. It proposes ROAP, a lightweight, architecture-agnostic pipeline that couples an Adaptive-XY-Gap Tree (AXG-Tree) for robust reading order extraction with a Reading-Order-Aware Relative Position Bias (RO-RPB) and a Textual-Token Sub-block Attention Prior (TT-Prior) to guide attention and suppress visual noise, respectively. Across FUNSD and CORD benchmarks, ROAP yields consistent gains for LayoutLMv3 and GeoLayoutLM in semantic entity recognition and relation extraction, demonstrating improved reading-flow reasoning and finer-grained text-text interactions without backbone modification. These results underscore the practical value of explicitly modeling reading order and modality-aware attention priors for robust, scalable VrDU.

Abstract

The efficacy of Multimodal Transformers in visually-rich document understanding (VrDU) is critically constrained by two inherent limitations: the lack of explicit modeling for logical reading order and the interference of visual tokens that dilutes attention on textual semantics. To address these challenges, this paper presents ROAP, a lightweight and architecture-agnostic pipeline designed to optimize attention distributions in Layout Transformers without altering their pre-trained backbones. The proposed pipeline first employs an Adaptive-XY-Gap (AXG-Tree) to robustly extract hierarchical reading sequences from complex layouts. These sequences are then integrated into the attention mechanism via a Reading-Order-Aware Relative Position Bias (RO-RPB). Furthermore, a Textual-Token Sub-block Attention Prior (TT-Prior) is introduced to adaptively suppress visual noise and enhance fine-grained text-text interactions. Extensive experiments on the FUNSD and CORD benchmarks demonstrate that ROAP consistently improves the performance of representative backbones, including LayoutLMv3 and GeoLayoutLM. These findings confirm that explicitly modeling reading logic and regulating modality interference are critical for robust document understanding, offering a scalable solution for complex layout analysis. The implementation code will be released at https://github.com/KevinYuLei/ROAP.

Figures (4)

• Figure 1: Overall architecture of the proposed ROAP pipline.
• Figure 2: Visualization of reading order generation on a sample from the FUNSD dataset. (a) The raw output from OCR engines often exhibits chaotic indices (e.g., the disjointed header and title regions), which disrupts semantic continuity. (b) The proposed AXG-Tree successfully reconstructs the human-perceived logical flow (indices $0\to 1\to \dots$), ensuring that spatially adjacent text blocks are grouped sequentially.
• Figure 3: Visualization of prediction results on a representative sample. The left side of each sub-figure shows the Baseline result, and the right side shows the ROAP-LayoutLMv3 result. Green: True Positive; Red: False Negative; Blue: False Positive; Purple: Wrong Entity Type. ROAP successfully corrects the semantic misclassification in the table region and reduces false alarms in the background.
• Figure 4: Comparison of attention maps for Layer 5, Head 5 on the sample. Left: Baseline LayoutLMv3 (Max val: 0.5796). Right: ROAP-LayoutLMv3 (Max val: 0.6766). Brighter spots indicate higher attention weights.