DvD: Unleashing a Generative Paradigm for Document Dewarping via Coordinates-based Diffusion Model

TL;DR

This paper tackles the challenge of preserving document structure during dewarping of photographed documents by reframing dewarping as a coordinates-based diffusion process. The proposed DvD model performs latent diffusion in a 64×64 coordinate space, guided by a rich compound conditioning scheme and a time-variant condition refinement to retain layout and textual content. A new large-scale AnyPhotoDoc6300 benchmark is introduced to enable fine-grained evaluation across domains and warping patterns, and extensive experiments show state-of-the-art performance on multiple benchmarks with reasonable efficiency. The work highlights a paradigm shift from discriminative regression to generative mapping generation for document dewarping, with broad implications for downstream OCR and document understanding tasks.

Abstract

Document dewarping aims to rectify deformations in photographic document images, thus improving text readability, which has attracted much attention and made great progress, but it is still challenging to preserve document structures. Given recent advances in diffusion models, it is natural for us to consider their potential applicability to document dewarping. However, it is far from straightforward to adopt diffusion models in document dewarping due to their unfaithful control on highly complex document images (e.g., 2000$times$3000 resolution). In this paper, we propose DvD, the first generative model to tackle document Dewarping via a Diffusion framework. To be specific, DvD introduces a coordinate-level denoising instead of typical pixel-level denoising, generating a mapping for deformation rectification. In addition, we further propose a time-variant condition refinement mechanism to enhance the preservation of document structures. In experiments, we find that current document dewarping benchmarks can not evaluate dewarping models comprehensively. To this end, we present AnyPhotoDoc6300, a rigorously designed large-scale document dewarping benchmark comprising 6,300 real image pairs across three distinct domains, enabling fine-grained evaluation of dewarping models. Comprehensive experiments demonstrate that our proposed DvD can achieve state-of-the-art performance with acceptable computational efficiency on multiple metrics across various benchmarks, including DocUNet, DIR300, and AnyPhotoDoc6300. The new benchmark and code will be publicly available at https://github.com/hanquansanren/DvD.

Figures (12)

• Figure 1: Comparisons to the existing paradigm for document dewarping. Unlike the existing paradigms (a) and (b) struggle with either precise structural preservation or faithful content generation, our DVD (c) can yield flat document images with precise yet faithful content through a mapping generation paradigm.
• Figure 2: Framework of the proposed DvD. Given a warped document as input, DvD generates latent variables $m$ to characterize the mapping for deformation rectification. We leverage a compound condition $c_t$: the features for raw document images $f_d$, document foreground mask $f_m$, text-lines $f_l$, and the time-variant condition of the intermediate result $r_t$, respectively. We visualize the reverse denoise generation process in the right pink region.
• Figure 3: General document dewarping pipeline. We propose a dewarping model to predict a backward mapping for deformation rectification, consisting of two coordinates channels.
• Figure 4: Detailed architectures of condition embedding blocks (CEB) and fusion generation blocks (FGB). We ignore some simple operations, such as cosine position encoding and activation functions.
• Figure 5: A sample data array visualization across 3 distinct domain spectra (A, B, C) in our AnyPhotoDoc6300 Benchmark. We aim to evaluate the dewarping capability of the model to different warping patterns (D) under well-distinguished domain combinations.