Reasoning-OCR: Can Large Multimodal Models Solve Complex Logical Reasoning Problems from OCR Cues?

TL;DR

This work introduces Reasoning-OCR, a benchmark crafted to assess complex logical reasoning driven by OCR cues across six visual contexts and six reasoning types, while minimizing the need for specialized domain knowledge. The dataset comprises 140 images and 150 English/Chinese questions with hints, designed to require multi-hop reasoning beyond standard OCR tasks. Evaluations on nine open-source LMMs and GPT-4o reveal that current models lag in complex OCR-based reasoning, with text-centric approaches particularly behind generic models; chain-of-thought prompts, hints, and task-specific instructions can yield meaningful gains. The study underscores the potential of OCR-informed reasoning and outlines directions for data-centric improvements and prompting strategies to advance LMM capabilities in real-world tasks such as planning and decision making.

Abstract

Large Multimodal Models (LMMs) have become increasingly versatile, accompanied by impressive Optical Character Recognition (OCR) related capabilities. Existing OCR-related benchmarks emphasize evaluating LMMs' abilities of relatively simple visual question answering, visual-text parsing, etc. However, the extent to which LMMs can deal with complex logical reasoning problems based on OCR cues is relatively unexplored. To this end, we introduce the Reasoning-OCR benchmark, which challenges LMMs to solve complex reasoning problems based on the cues that can be extracted from rich visual-text. Reasoning-OCR covers six visual scenarios and encompasses 150 meticulously designed questions categorized into six reasoning challenges. Additionally, Reasoning-OCR minimizes the impact of field-specialized knowledge. Our evaluation offers some insights for proprietary and open-source LMMs in different reasoning challenges, underscoring the urgent to improve the reasoning performance. We hope Reasoning-OCR can inspire and facilitate future research on enhancing complex reasoning ability based on OCR cues. Reasoning-OCR is publicly available at https://github.com/Hxyz-123/ReasoningOCR.

Figures (13)

• Figure 1: Samples in Reasoning-OCR. Relevant textual cues in the image are highlighted with red circles or rectangles for clarity, which are not visible to LMMs. Our questions demonstrate higher reasoning complexity compared to the ones in source datasets.
• Figure 2: The visual scenario distribution in Reasoning-OCR. The collected images cover six visual scenarios, including chart, product label, document, natural image, screen shot, and token (from the most to the least).
• Figure 3: The distribution of question types. (a) shows the distribution of the six question types across the Reasoning-OCR while (b) describes the distribution of the six question types within each of the data sources.
• Figure 4: Visualization of LMMs' responses on Reasoning-OCR under CoT and task-specific instruction settings. Relevant textual cues in the image are highlighted with red circles or rectangles for clarity, which are not visible to LMMs. The key elements of the question are emphasized in blue. In the responses generated by the LMMs, incorrect reasoning steps are marked in red, while correct reasoning steps are indicated in green.
• Figure 5: Failure cases of GPT-4o on Reasoning-OCR. Relevant textual cues in the images are highlighted with red circles or rectangles for clarity, which are not visible to LMMs. The key elements in questions are emphasized in blue. In LMMs' responses, incorrect reasoning steps are marked in red.