LOCR: Location-Guided Transformer for Optical Character Recognition

TL;DR

LOCR presents a location-guided Transformer that jointly predicts tokens and the position of the next token during autoregression, enabling end-to-end OCR for academically cluttered documents. A 2D positional encoding and a prompt module fuse visual and spatial cues, while accumulation and blank decay mitigate repetitive degeneration, and an interactive OCR mode enables human-guided corrections on difficult layouts. Trained on a large-scale data engine with 125K pages and 77M text-location pairs, LOCR achieves state-of-the-art performance on the arXiv test set across edit distance, BLEU, METEOR, and F-measure, and substantially reduces repetition in both in-domain and out-of-domain data. The work also provides a data engine and invites dataset release to foster research in location-aware OCR and domain-specific data construction for robust document digitization and LLM training workflows.

Abstract

Academic documents are packed with texts, equations, tables, and figures, requiring comprehensive understanding for accurate Optical Character Recognition (OCR). While end-to-end OCR methods offer improved accuracy over layout-based approaches, they often grapple with significant repetition issues, especially with complex layouts in Out-Of-Domain (OOD) documents.To tackle this issue, we propose LOCR, a model that integrates location guiding into the transformer architecture during autoregression. We train the model on a dataset comprising over 77M text-location pairs from 125K academic document pages, including bounding boxes for words, tables and mathematical symbols. LOCR adeptly handles various formatting elements and generates content in Markdown language. It outperforms all existing methods in our test set constructed from arXiv, as measured by edit distance, BLEU, METEOR and F-measure.LOCR also reduces repetition frequency from 4.4% of pages to 0.5% in the arXiv dataset, from 13.2% to 1.3% in OOD quantum physics documents and from 8.1% to 1.8% in OOD marketing documents. Additionally, LOCR features an interactive OCR mode, facilitating the generation of complex documents through a few location prompts from human.

Figures (12)

• Figure 1: An overview of three components of our work: a large-scale dataset with positional annotation and a data engine, a location-guided OCR model for various layouts, and an interactive mode for humans to prompt the model and modify data collection.
• Figure 2: Data Processing. Step1: Add a unique RGB identifier to each word by parsing the Tex file. Step2: Convert source file into Markdown and PDF formats respectively. Step3: Extract color-bbox pairs from colored PDF, color-text pairs from Markdown, and merge the two to get the text-bbox pairs.
• Figure 3: Model Architecture. Left: Image encoder and decoder of transformer structure. Right: Position detection head and token projection. Purple: Prompt module consisting of positional encodings and position detection head. Red: Interactive mode with human-reviewed input.
• Figure 4: Examples of our model output. Left: Origin image of document page. Right: Model output converted to Markdown and rendered back into a PDF. More detailed examples are available in Appendix \ref{['appendix:examples']}
• Figure A1: Dataset example. Bounding boxes of texts are highlighted in pink, mathematical expressions in blue, and tables in green.