GutenOCR: A Grounded Vision-Language Front-End for Documents

TL;DR

GutenOCR reframes OCR as a grounded front-end by fine-tuning Qwen2.5-VL backbones into a single 3B/7B checkpoint that supports full-page reading, detection, localized reading, and conditional detection via a prompt-based interface. It combines real-world and synthetic data in a curriculum to achieve long-context grounding, returning either plain transcripts or structured line/paragraph outputs with bounding boxes. The paper defines a comprehensive evaluation protocol and demonstrates substantial gains in in-domain and Fox benchmarks for grounding and region-level reading, while highlighting trade-offs in page-level linearization and formula recognition. The work provides an open training recipe and model weights, offering a practical, verifiable front-end for document extraction and RAG systems and advancing toward richer document hologram representations for evidence-based QA.

Abstract

GutenOCR is a family of grounded OCR front-ends obtained by fine-tuning Qwen2.5-VL-3B and Qwen2.5-VL-7B. The resulting single-checkpoint vision-language models expose reading, detection, and grounding through a unified, prompt-based interface. Trained on business documents, scientific articles, and synthetic grounding data, the models support full-page and localized reading with line- and paragraph-level bounding boxes and conditional ``where is x?'' queries. We introduce a grounded OCR evaluation protocol and show that GutenOCR-7B more than doubles the composite grounded OCR score of its Qwen2.5-VL-7B backbone on 10.5K held-out business and scientific pages (0.40 to 0.82). On Fox and OmniDocBench v1.5, our approach substantially improves region- and line-level OCR as well as text-detection recall, but reveals trade-offs in page-level linearization, color-guided OCR, and formula-heavy layouts.

Figures (7)

• Figure 1: Capability profile of GutenOCR-3B/7B, their Qwen2.5-VL backbones, and Qwen2.5-based OCR baselines on nine grounded OCR tasks spanning our in-domain suite and Fox. Each spoke shows a raw score in $[0,1]$ (higher is better): in-domain page-, line-, and local-reading tasks are scored as $1-\mathrm{CER}$, full and conditional detection as F1, and Fox page-, region-, line-, and color-guided OCR as $1-\mathrm{CER}$. GutenOCR substantially improves detection, fine-grained reading, and Fox region/line OCR, while trading off some page-level and color-guided performance relative to OCR-specialized baselines.
• Figure 2: Unified GutenOCR interface. A single vision--language model consumes a page image, optionally conditioned on a text query $q$ or bounding box $b$, and serves multiple OCR-style tasks (reading, detection, conditional detection, localized reading) through prompt-specified input and output schemas. Sub-panels depict each task family: full page detection (top-left), full page reading (top-right), local reading (bottom-right), and conditional detection (bottom-left). Sample page from Zhou2010BCH.
• Figure 3: Example of the layout-sensitive text2d representation. Long runs of spaces encode horizontal alignment (e.g., right-justified page number), while blank lines encode vertical gaps between sections and between the main text and the flow diagram. Page from Dubayova2010FearDelay.
• Figure 4: Qualitative comparison of Qwen2.5-VL-3B and GutenOCR-3B showing complementary strengths and weaknesses. (a) GutenOCR excels at region extraction (CER: 0.004 vs. 0.987). (b) Region training transfers to line pointer tasks (CER: 0.000 vs. 0.988). (c) GutenOCR suffers catastrophic forgetting on color tasks (CER: 0.997 vs. 0.000). (d) Trade-off: GutenOCR achieves higher content F1 (0.983 vs. 0.597) despite worse reading order (CER: 0.687 vs. 0.574) relative to Fox’s reference expectations.
• Figure 5: Qualitative OmniDocBench text-detection example. Red boxes denote annotated text spans; blue boxes denote GutenOCR predictions on the same page. Many blue boxes fall on readable text (e.g., table cells and decorative labels) that OmniDocBench does not label as text, so non-overlapping predictions cannot be reliably treated as false positives. This motivates our recall-only evaluation protocol in Section \ref{['sec:omnidoc-text-detection']}.