Document Image Rectification Bases on Self-Adaptive Multitask Fusion

TL;DR

This paper tackles distorted document image rectification by exploiting multi-granular cues across global and local features. It introduces SalmRec, an end-to-end framework that uses inter-task feature aggregation and a gating mechanism to dynamically balance global (3D coordinates, UV maps) and local (horizontal/vertical lines) information, with a Transformer decoder predicting the 2D deformation field. The approach achieves state-of-the-art results on DIR300, DocReal, and DocUNet benchmarks and improves OCR metrics (ED/CER) through effective cross-task fusion. Ablation studies confirm that the proposed FA and gating components mitigate negative interference among tasks while enhancing feature complementarity. Overall, SalmRec demonstrates that adaptive, cross-task collaboration among auxiliary features can significantly improve document rectification and downstream recognition in real-world scenarios.

Abstract

Deformed document image rectification is essential for real-world document understanding tasks, such as layout analysis and text recognition. However, current multi-task methods -- such as background removal, 3D coordinate prediction, and text line segmentation -- often overlook the complementary features between tasks and their interactions. To address this gap, we propose a self-adaptive learnable multi-task fusion rectification network named SalmRec. This network incorporates an inter-task feature aggregation module that adaptively improves the perception of geometric distortions, enhances feature complementarity, and reduces negative interference. We also introduce a gating mechanism to balance features both within global tasks and between local tasks effectively. Experimental results on two English benchmarks (DIR300 and DocUNet) and one Chinese benchmark (DocReal) demonstrate that our method significantly improves rectification performance. Ablation studies further highlight the positive impact of different tasks on dewarping and the effectiveness of our proposed module.

Figures (7)

• Figure 1: Overview architecture of our proposed SalmRec. For a given distorted image, SalmRec learns to predict a 2D deformation field by multi-decoder segmentation module ($D_i$), inter-task feature aggregation ($FA$), gating mechanisms ($G_i$), and Transformer decoder. $\boldsymbol{F}$ represents the output feature map.
• Figure 2: Inter-task Feature Aggregation (FA) Module. The four tasks are divided into four groups, where $\boldsymbol{F}_{3D}$ is fused with the global attention score obtained from the other three tasks for feature interconnection aggregation.
• Figure 3: Gating Mechanism. $\boldsymbol{F}_a$ is the universal feature extracted from the original distorted image by the encoder of the segmentation module. $\boldsymbol{F}^{\prime}_{3D}$ and $\boldsymbol{F}^{\prime}_{uv}$ are two aggregated features with global properties. Similar operations are performed on local features $\boldsymbol{F}^{\prime}_{hline}$ and $\boldsymbol{F}^{\prime}_{vline}$. $C$, $H$, $W$ represents the channel, height and width of the feature map respectively.
• Figure 4: Qualitative visual comparison with existing methods on DIR300 benchmark Feng2022GeometricRL. The first column "Distorted" means the given distorted document image, and the last column "Ground-Truth" is the flattened reference image.
• Figure 5: Qualitative visual comparison with existing methods on DocReal benchmark Yu_2024_WACV.