UVDoc: Neural Grid-based Document Unwarping

TL;DR

This paper proposes a novel method for grid-based single-image document unwarping via a fully convolutional deep neural network that learns to predict the 3D grid mesh of the document and the corresponding 2D unwarping grid in a dual-task fashion, and proposes a metric that quantifies line straightness after unwarping.

Abstract

Restoring the original, flat appearance of a printed document from casual photographs of bent and wrinkled pages is a common everyday problem. In this paper we propose a novel method for grid-based single-image document unwarping. Our method performs geometric distortion correction via a fully convolutional deep neural network that learns to predict the 3D grid mesh of the document and the corresponding 2D unwarping grid in a dual-task fashion, implicitly encoding the coupling between the shape of a 3D piece of paper and its 2D image. In order to allow unwarping models to train on data that is more realistic in appearance than the commonly used synthetic Doc3D dataset, we create and publish our own dataset, called UVDoc, which combines pseudo-photorealistic document images with physically accurate 3D shape and unwarping function annotations. Our dataset is labeled with all the information necessary to train our unwarping network, without having to engineer separate loss functions that can deal with the lack of ground-truth typically found in document in the wild datasets. We perform an in-depth evaluation that demonstrates that with the inclusion of our novel pseudo-photorealistic dataset, our relatively small network architecture achieves state-of-the-art results on the DocUNet benchmark. We show that the pseudo-photorealistic nature of our UVDoc dataset allows for new and better evaluation methods, such as lighting-corrected MS-SSIM. We provide a novel benchmark dataset that facilitates such evaluations, and propose a metric that quantifies line straightness after unwarping. Our code, results and UVDoc dataset are available at https://github.com/tanguymagne/UVDoc.

Figures (10)

• Figure 1: An overview of our data capture setup and sample data acquired in the process. The top shows our capture setup: [1] UV lights, [2] SR305 depth camera, [3] deformed sheet of paper, [4] regular light. The bottom shows a capture sample including RGB images of the normally lit and UV-lit paper, and its depth image.
• Figure 2: The pipeline used to create a sample of our UVDoc dataset. It combines the captured image of a blank paper, the texture and the background.
• Figure 3: Our unwarping pipeline. We start with an RGB image of a warped document and feed it into our encoder-style network. The network predicts both a 3D grid mesh (top branch), as well as a 2D unwarping grid (bottom branch) in parallel. The 2D unwarping grid is then bilinearly interpolated to the desired output image resolution and is used to sample pixels from the input image to obtain the final unwarped document image.
• Figure 4: The shaded and unshaded version of a sample from the UVDoc benchmark, identical up to shading. The shaded version and unshaded version have a CER of 0.439 and 0.004 respectively and ED of 959 and 14. Note that the unshaded version has non-zero CER and ED as it is compared to the original texture, while it has been warped and unwarped using our coarse bilinearly interpolated 2D grid, and thus includes some artifacts.
• Figure 5: Our new horizontal line metric is the standard deviation of the $y$ coordinate of warped horizontal lines (middle) unwarped using the predicted backward mapping (right in red, ground-truth in black).