LogicOCR: Do Your Large Multimodal Models Excel at Logical Reasoning on Text-Rich Images?

TL;DR

LogicOCR tackles the challenge of evaluating large multimodal models on complex logical reasoning over text-rich images by introducing two complementary subdatasets, LogicOCR-Gen and LogicOCR-Real, and a scalable image-generation pipeline based on GPT-Image-1 driven by a Chinese civil service exam corpus. The study reveals that Chain-of-Thought prompting often fails to consistently improve multimodal reasoning on these tasks, while test-time scaling and OCR robustness remain important factors; to address perceptual gaps, it proposes TextCue, a training-free cropping method leveraging relative attention and a text-segmentation model to amplify important textual regions. Across a broad spectrum of models, results show that multimodal reasoning performance lags behind text-only reasoning, highlighting a persistent gap between visual reading and logical deduction, even as OCR capabilities are strong. The work also demonstrates that TextCue can provide measurable gains for several LMMs and offers a practical direction for improving multimodal reasoning without additional training. Overall, LogicOCR serves as a valuable resource for diagnosing and guiding improvements in the integration of reading and reasoning in large multimodal systems.

Abstract

Recent advances in Large Multimodal Models (LMMs) have revolutionized their reasoning and Optical Character Recognition (OCR) capabilities. However, their complex logical reasoning performance on text-rich images remains underexplored. To bridge this gap, we introduce LogicOCR, a benchmark comprising 2780 questions with two subsets, i.e., LogicOCR-Gen with 1100 multi-choice questions on generated images, and LogicOCR-Real with 1680 meticulously designed free-form questions on real-world images. For constructing LogicOCR-Gen, we first curate a text corpus from the Chinese National Civil Servant Examination, and customize an automatic pipeline to steer GPT-Image-1 to generate images with varied layouts and fonts, ensuring contextual relevance and visual realism. Then, the generated images are manually verified. We evaluate a range of representative LMMs under Chain-of-Thought (CoT) and direct-answer settings. Our multi-dimensional analysis reveals key insights, such as the impact of test-time scaling, input modality differences, and sensitivity to visual-text orientation. Notably, LMMs still lag in multimodal reasoning compared to text-only inputs, indicating that they have not fully bridged visual reading with reasoning. Moreover, we propose TextCue, a training-free method that enhances LMMs' perception of image regions containing important text cues for solving questions. We leverage LMMs' attention maps and an off-the-shelf text segmentation specialist to determine the region, which is then cropped and enlarged to augment the original image. Experiments show its effectiveness, e.g., a 1.8% accuracy gain over LLaVA-OV-1.5-8B under the CoT setting. Our benchmark is available at https://github.com/MiliLab/LogicOCR.

Figures (26)

• Figure 1: Illustration of the LogicOCR-Gen data construction process and sample images, showcasing background-style and text-illustration interleaved layouts from top to bottom.
• Figure 2: Comparison of the newly crafted multi-hop reasoning questions in LogicOCR-Real and the original questions in previous benchmarks.
• Figure 3: Illustration of TextCue, with visual cropping (left) and answering (right) stages. First, a relative attention mechanism is used to achieve relative attention maps. We introduce an adaptive layer selection scheme to determine the most salient attention map. Then, we convert the selected attention map into a rough box region, which is subsequently refined with the predictions of an off-the-shelf text segmentation model. Given the box region, a local patch is cropped, enlarged, and finally combined with the original image and question for answering.
• Figure 4: Comparison of LMMs on LogicOCR under CoT and direct answering settings. Applying TextCue on LLaVA-OV-1.5-4B achieves 0.1% and 0.8% accuracy improvements under CoT and direct answering settings, respectively, while achieving 1.8% and 0.7% accuracy gains on LLaVA-OV-1.5-8B under CoT and direct answering settings.
• Figure 5: Comparison of average accuracy and output length (completion tokens) between general LMMs and their reasoning-enhanced counterparts. Notably, o4-mini achieves much higher accuracy using fewer tokens than QvQ-72B-Preview.