Ocean-OCR: Towards General OCR Application via a Vision-Language Model

TL;DR

Ocean-OCR introduces a 3B multimodal model with Native Resolution ViT to handle arbitrary image resolutions and a dedicated OCR-centric data strategy, achieving state-of-the-art OCR performance while preserving general multimodal understanding. The approach combines a NaViT-based visual encoder, an MLP projector, and a Qwen-2.5-3B LLM, trained through a three-stage pipeline on diverse pure-text, image-text, caption, and OCR data, including synthetic augmentation. Empirical results show Ocean-OCR surpassing specialized OCR models like TextIn and PaddleOCR across DocVQA, TextVQA, ChartQA, and OCRBench, and strong general benchmark performance, validating its practicality for real-world OCR tasks. The work highlights the importance of high-quality OCR data, dynamic resolution processing, and targeted fine-tuning to bridge the gap between OCR-specific accuracy and broad multimodal understanding.

Abstract

Multimodal large language models (MLLMs) have shown impressive capabilities across various domains, excelling in processing and understanding information from multiple modalities. Despite the rapid progress made previously, insufficient OCR ability hinders MLLMs from excelling in text-related tasks. In this paper, we present \textbf{Ocean-OCR}, a 3B MLLM with state-of-the-art performance on various OCR scenarios and comparable understanding ability on general tasks. We employ Native Resolution ViT to enable variable resolution input and utilize a substantial collection of high-quality OCR datasets to enhance the model performance. We demonstrate the superiority of Ocean-OCR through comprehensive experiments on open-source OCR benchmarks and across various OCR scenarios. These scenarios encompass document understanding, scene text recognition, and handwritten recognition, highlighting the robust OCR capabilities of Ocean-OCR. Note that Ocean-OCR is the first MLLM to outperform professional OCR models such as TextIn and PaddleOCR.

Figures (6)

• Figure 1: Comparison with models across various OCR scenarios and benchmarks.(Left) Current mainstream MLLMs and specific OCR models across multiple noteworthy OCR abilities, that is, scene-level, document-level, and handwritten-level text recognition. E, F, P, R, B, and M are the abbreviations for Edit Distance, F1-Score, Precision, Recall, BLEU, and METEOR respectively. For Edit Distance, the plotted score is computed with $x_{after} = 100 - x_{before}$ for better visualization. (Right) Comparison of mainstream MLLMs performance on OCR benchmarks.
• Figure 2: Overview of Ocean-OCR-3B. Following most of current MLLMs qwen2llava_nextliu2024points1, Ocean-OCR-3B uses the conventional LLaVA-style structure that consists of a vision encoder, a MLP projector, and a LLM. To better support native dynamic high resolution in various OCR scenarios, we use NaViT-style navit vision encoder.
• Figure 3: Strong OCR ability of Ocean-OCR-3B. Our model shows strong text recognition ability across various real-world scenarios. We simply use What is written in this image? as prompt.
• Figure 4: Strong OCR ability of Ocean-OCR-3B. Our model shows strong ability for handwritten text recognition in Chinese and English. We simply use Please extract all texts in this image. as prompt.
• Figure 5: Strong OCR ability of Ocean-OCR-3B. Our model shows strong ability for PDF document text recognition in Chinese and English. We simply use Please extract all texts in this image. as prompt.