Exploring OCR Capabilities of GPT-4V(ision) : A Quantitative and In-depth Evaluation

TL;DR

This study provides the first rigorous quantitative assessment of GPT-4V(ision) across core OCR tasks, including STR, HTR, HMER, TSR, and VIE, using standard benchmarks and carefully designed prompts. It reveals strong performance on Latin and English text but notable weaknesses with multilingual content, complex layouts, and fine-grained symbol recognition, underscoring that general LMMs do not yet surpass specialized OCR models. The findings highlight the trade-offs between broad multimodal capabilities and task-specific accuracy, and suggest avenues such as semantic enhancement, downstream fine-tuning, and autonomous data creation to better harness LMMs for OCR. The work offers a critical reference for researchers and practitioners aiming to integrate LMMs into OCR pipelines while maintaining emphasis on specialized models where needed, and it lays groundwork for ongoing evaluation updates as models evolve.

Abstract

This paper presents a comprehensive evaluation of the Optical Character Recognition (OCR) capabilities of the recently released GPT-4V(ision), a Large Multimodal Model (LMM). We assess the model's performance across a range of OCR tasks, including scene text recognition, handwritten text recognition, handwritten mathematical expression recognition, table structure recognition, and information extraction from visually-rich document. The evaluation reveals that GPT-4V performs well in recognizing and understanding Latin contents, but struggles with multilingual scenarios and complex tasks. Specifically, it showed limitations when dealing with non-Latin languages and complex tasks such as handwriting mathematical expression recognition, table structure recognition, and end-to-end semantic entity recognition and pair extraction from document image. Based on these observations, we affirm the necessity and continued research value of specialized OCR models. In general, despite its versatility in handling diverse OCR tasks, GPT-4V does not outperform existing state-of-the-art OCR models. How to fully utilize pre-trained general-purpose LMMs such as GPT-4V for OCR downstream tasks remains an open problem. The study offers a critical reference for future research in OCR with LMMs. Evaluation pipeline and results are available at https://github.com/SCUT-DLVCLab/GPT-4V_OCR.

Figures (7)

• Figure 1: Illustration of word-level scene text recognition. In the answers of GPT-4V, we highlight the characters that match the GT in green and characters that do not match in red. GPT-4V can recognize curved, slanted, and artistic English text, while common-style Chinese text can not be recognized.
• Figure 2: Illustration of handwritten text recognition. (a), (b), (c), and (d) are samples of page-level IAM, line-level IAM, page-level CASIA-HWDB, and line-level CASIA-HWDB, respectively. In the responses of GPT-4V, we highlight characters that match the GT in green and characters that do not match in red. For English text, GPT-4V demonstrates excellent performance. In contrast, for Chinese text, GPT-4V has generated a passage of text that is semantically coherent, but it is not associated with the ground truth text (GT).
• Figure 3: Illustration of handwritten mathematical expression recognition. In each example, the left side displays the input image, while the right side shows the image rendered from the LaTeX sequence output by GPT-4V. In the answer of GPT-4V, we highlight elements that match the GT in green and elements that do not match in red. The symbol _ in red represents the missing elements in the output.
• Figure 4: Illustration of table structure recognition. (a) and (c) are two input images, (b) and (d) are the corresponding visualized images of GPT-4V's html-style output sequence. (e) is the output sequence of (c), where the elements that GPT-4V indicate the omitted content are highlighted in red.
• Figure 5: Illustration of error cases of the SER task. The text content enclosed within the red box is incorrectly identified as header entities.