Handheld Video Document Scanning: A Robust On-Device Model for Multi-Page Document Scanning

TL;DR

An efficient, on-device deep learning model that is accurate and robust for handheld scanning, a novel data collection and annotation technique for video document scanning, and state-of-the-art results on the PUCIT page turn dataset are proposed.

Abstract

Document capture applications on smartphones have emerged as popular tools for digitizing documents. For many individuals, capturing documents with their smartphones is more convenient than using dedicated photocopiers or scanners, even if the quality of digitization is lower. However, using a smartphone for digitization can become excessively time-consuming and tedious when a user needs to digitize a document with multiple pages. In this work, we propose a novel approach to automatically scan multi-page documents from a video stream as the user turns through the pages of the document. Unlike previous methods that required constrained settings such as mounting the phone on a tripod, our technique is designed to allow the user to hold the phone in their hand. Our technique is trained to be robust to the motion and instability inherent in handheld scanning. Our primary contributions in this work include: (1) an efficient, on-device deep learning model that is accurate and robust for handheld scanning, (2) a novel data collection and annotation technique for video document scanning, and (3) state-of-the-art results on the PUCIT page turn dataset.

Figures (7)

• Figure 1: Sample video frames during an automatic video capture and model predictions. After capturing the left page, the user pans to the right page. The model detects the change and waits for a frame to detect problems such as the user's hand over the page content. As the user turns the page, the model detects the change and waits to capture.
• Figure 2: Our Mechanical Turk annotation interface. Most videos could be annotated in a single playthrough of the video by clicking the "m" hotkey to mark page changes while the video was playing. If needed, the annotations could be quickly adjusted using hotkeys.
• Figure 3: Example frames for each of the CapE attributes labeled in the dataset.
• Figure 4: Overview of our proposed system: On the first frame, because the frame is in the middle of a PCE, the CapN is not run. On the second frame, the PCN predicts it is not a PCE and likely a CapE, so the slower but more accurate CapN runs.
• Figure 5: Visualization of look-ahead frames and processing multiple frames as input for the PCN. The CapE output uses the label of the current frame while the PCE uses the label from three frames in the past (represented with the dotted lines).