OCRVerse: Towards Holistic OCR in End-to-End Vision-Language Models

TL;DR

OCRVerse introduces the first end-to-end holistic OCR system that unifies text-centric and vision-centric recognition. It combines comprehensive multi-domain data engineering with a two-stage SFT-RL training regime to learn cross-domain representations and then tailor domain-specific outputs via personalized rewards. The approach achieves strong results across both text-centric benchmarks like OmniDocBench and vision-centric benchmarks such as ChartMimic, UniSVG, and Image2LaTeX-plot, demonstrating competitive performance with significantly fewer parameters than many open- and closed-source models. This holistic OCR framework enables robust understanding of visually information-dense content, with practical impact on data visualization, web page analysis, and scientific figure interpretation. The work highlights a scalable path toward unified multimodal OCR and paves the way for broader adoption in multimodal AI systems.

Abstract

The development of large vision language models drives the demand for managing, and applying massive amounts of multimodal data, making OCR technology, which extracts information from visual images, increasingly popular. However, existing OCR methods primarily focus on recognizing text elements from images or scanned documents (\textbf{Text-centric OCR}), neglecting the identification of visual elements from visually information-dense image sources (\textbf{Vision-centric OCR}), such as charts, web pages and science plots. In reality, these visually information-dense images are widespread on the internet and have significant real-world application value, such as data visualization and web page analysis. In this technical report, we propose \textbf{OCRVerse}, the first holistic OCR method in end-to-end manner that enables unified text-centric OCR and vision-centric OCR. To this end, we constructe comprehensive data engineering to cover a wide range of text-centric documents, such as newspapers, magazines and books, as well as vision-centric rendered composites, including charts, web pages and scientific plots. Moreover, we propose a two-stage SFT-RL multi-domain training method for OCRVerse. SFT directly mixes cross-domain data to train and establish initial domain knowledge, while RL focuses on designing personalized reward strategies for the characteristics of each domain. Specifically, since different domains require various output formats and expected outputs, we provide sufficient flexibility in the RL stage to customize flexible reward signals for each domain, thereby improving cross-domain fusion and avoiding data conflicts. Experimental results demonstrate the effectiveness of OCRVerse, achieving competitive results across text-centric and vision-centric data types, even comparable to large-scale open-source and closed-source models.

Figures (4)

• Figure 1: Performance comparison of OCRVerse on text-centric OCR tasks (top row) and vision-centric OCR tasks (bottom row). Since existing OCR methods primarily focus on text-centric scenarios, we compare against both specialized OCR models and general-purpose models for text-centric benchmarks, while comparing only against general-purpose models for vision-centric benchmarks.
• Figure 2: Comprehensive data coverage of OCRVerse for holistic OCR. Left (Text-centric data): Nine document scenarios including natural scenes, books, magazines, papers, reports, slides, exam papers, notes, and newspapers, covering high-frequency text scenarios in daily life. Right (Vision-centric data): Six specialized scenarios including charts, webpages, icons, geometry, circuits, and molecules, focusing on professional structured content that requires code-level representations.
• Figure 3: Multi-stage data construction pipeline integrating text-centric and vision-centric sources. Text-centric pipeline (top) processes open-source data, real-world PDFs, and synthetic data through cleaning and VLM-based annotation. Vision-centric pipeline (bottom) collects chart, webpage, and SVG data, etc., applies quality filtering, and generates annotations through visualization rendering and structure extraction.
• Figure 4: OCRVerse training pipeline. Stage 1: SFT with unified cross-domain data. Stage 2: RL with domain-specific data construction and personalized reward mechanisms for text-centric (rule-based) and vision-centric (visual fidelity) optimization.