RoDLA: Benchmarking the Robustness of Document Layout Analysis Models

TL;DR

The paper tackles the lack of robustness evaluation in Document Layout Analysis by introducing a large-scale perturbation benchmark (approximately 450k images across PubLayNet-P, DocLayNet-P, and M$^6$Doc-P) with 12 perturbation types at 3 severity levels, plus two metrics, Mean Perturbation Effect ($mPE$) and Mean Robustness Degradation ($mRD$). It proposes RoDLA, a DLA model that combines channel-attention and average-pooling strategies within an InternImage/Transformer-based backbone to extract perturbation-insensitive features, achieving state-of-the-art robustness while preserving strong clean-data performance. Across datasets, RoDLA reaches superior $mRD$ scores (e.g., 115.7, 135.4, 150.4) and sizable $mAP$ gains (+3.8%, +7.1%, +12.1%), demonstrating the practical value of robust DLA in real-world document processing. The work provides a solid benchmark and design principles for robust document understanding, highlighting the importance of separating image perturbation effects from model robustness and guiding future developments in robust, real-world DLA systems.

Abstract

Before developing a Document Layout Analysis (DLA) model in real-world applications, conducting comprehensive robustness testing is essential. However, the robustness of DLA models remains underexplored in the literature. To address this, we are the first to introduce a robustness benchmark for DLA models, which includes 450K document images of three datasets. To cover realistic corruptions, we propose a perturbation taxonomy with 36 common document perturbations inspired by real-world document processing. Additionally, to better understand document perturbation impacts, we propose two metrics, Mean Perturbation Effect (mPE) for perturbation assessment and Mean Robustness Degradation (mRD) for robustness evaluation. Furthermore, we introduce a self-titled model, i.e., Robust Document Layout Analyzer (RoDLA), which improves attention mechanisms to boost extraction of robust features. Experiments on the proposed benchmarks (PubLayNet-P, DocLayNet-P, and M$^6$Doc-P) demonstrate that RoDLA obtains state-of-the-art mRD scores of 115.7, 135.4, and 150.4, respectively. Compared to previous methods, RoDLA achieves notable improvements in mAP of +3.8%, +7.1% and +12.1%, respectively.

Figures (7)

• Figure 1: [regular] Robust Document Layout Analysis (RoDLA) with hierarchical perturbations. (a) For benchmarking, we propose 5 groups (i.e., spatial transformation, content interference, inconsistency distortion, blur, and noise) and 12 types of perturbations (P1--P12) inspired by real-world document processing, as well as 3 severity levels (L1--L3) for each perturbation. Our RoDLA method obtains (b) higher mean Robust Degradation (mRD) in all 5 groups of perturbations, and (c) stable mAP scores across 3 levels of perturbation (e.g., defocus).
• Figure 2: Visualization of document perturbations.
• Figure 3: Analysis of perturbation evaluation metrics. (a) Comparison of perturbation metrics, including MS-SSIM, CW-SSIM, Degradation w.r.t a baseline, and the proposed mean Perturbation Effect (mPE). mPE is more balanced and inclusive to different perturbations. (b) Six documents perturbed by warping and defocus and their scores indicate that mPE is more sensitive to measure different levels.
• Figure 4: The architecture of RoDLA model. RoDLA is comprised of Encoder, Query Selection, Decoder, and Matching components. It optimizes the attention mechanism in Encoder, heightening focus on crucial tokens and reinforcing key token connections in multi-scale features to extract stable features.
• Figure 5: Comparison between perturbation evaluation metrics on 12 perturbation categories and 3 severity levels, including Image Quality Assessment methods (MS-SSIM and CW-SSIM), Degradation w.r.t a baseline, and the proposed mean Perturbation Effect (mPE). Other metrics cannot assess specific perturbations, for example MS-SSIM is insensitive to warping perturbation, and Degradation inversely measures texture perturbation across levels. In contrast, mPE is more balanced and inclusive to all perturbations and severity levels.