Words as Geometric Features: Estimating Homography using Optical Character Recognition as Compressed Image Representation

TL;DR

Addresses document alignment under privacy/storage constraints by estimating the homography via OCR-derived features, where the relationship between point coordinates obeys $ \mathbf{p'} = \mathbf{H} \mathbf{p} $ with $ \mathbf{H} \in \mathbb{R}^{3\times3} $ and 8 degrees of freedom. The method replaces gradient-based keypoints and descriptors with OCR word-centroids and text strings, using bipartite text matching followed by robust RANSAC to compute the homography. Across a set of test documents, the OCR-based approach often matches or exceeds the accuracy of image-based SIFT methods while using orders of magnitude less data. The approach enables privacy-preserving, scalable document registration with practical impact on automated form processing and verification.

Abstract

Document alignment and registration play a crucial role in numerous real-world applications, such as automated form processing, anomaly detection, and workflow automation. Traditional methods for document alignment rely on image-based features like keypoints, edges, and textures to estimate geometric transformations, such as homographies. However, these approaches often require access to the original document images, which may not always be available due to privacy, storage, or transmission constraints. This paper introduces a novel approach that leverages Optical Character Recognition (OCR) outputs as features for homography estimation. By utilizing the spatial positions and textual content of OCR-detected words, our method enables document alignment without relying on pixel-level image data. This technique is particularly valuable in scenarios where only OCR outputs are accessible. Furthermore, the method is robust to OCR noise, incorporating RANSAC to handle outliers and inaccuracies in the OCR data. On a set of test documents, we demonstrate that our OCR-based approach even performs more accurately than traditional image-based methods, offering a more efficient and scalable solution for document registration tasks. The proposed method facilitates applications in document processing, all while reducing reliance on high-dimensional image data.

Figures (6)

• Figure 1: SIFT features (top) and OCR features (bottom) on a document. Stated as a variant of Marr's definition of vision, here we introduce the idea that "knowing what word is where by looking", as captured by current and imperfect OCR models, is sufficient for the estimation of homography matrices relating the geometric transforms between pairs of documents for purposes of alignment for downstream vision tasks.
• Figure 2: Example of documents before image alignment through homography estimation. Corresponding fields are in different locations, confounding document analysis. These samples correspond to the court document (digital and scanned), flyer (standard scanned and perspective), and web printout (digital and scanned) described in the Experimental Evaluation section.
• Figure 3: SIFT matches (top) vs. OCR matches (bottom) for two sample pairs of documents. There are many spurious SIFT matches, though this is not too problematic given the robustness of RANSAC estimation techniques. However, the strengths of OCR-driven features are clear by comparison.
• Figure 4: SIFT matches (top) vs. OCR matches (bottom) for the passport card document described in the Experimental Evaluation section. The SIFT method creates many incorrect crossings, which eventually lead to an incorrect estimate of H.
• Figure 5: SIFT matches (top three images) vs. OCR matches (bottom) for sample pairs of the 'flyer' document. Gray images are 200dpi scans, and for our experiments we evaluate at both a direct and indirect perspective in multiple combinations. We note the presence of confounding matches in the SIFT features, compared to a clear relationship on the OCR features despite the relatively limited number of words available.