TextCoT: Zoom In for Enhanced Multimodal Text-Rich Image Understanding

TL;DR

TextCoT tackles the challenge of understanding text-rich images with high resolution by introducing a three-stage Chain-of-Thought framework that uses captioning for global context and grounding for local regions. The method progressively localizes answer regions and then inspects them with global context to improve QA accuracy, without requiring extra training. Experiments across multiple LMMs and diverse datasets show consistent improvements, highlighting strong generalization and plug-and-play practicality. This work advances multimodal reasoning by integrating global textual descriptions with localized visual cues for text-centric visual tasks.

Abstract

The advent of Large Multimodal Models (LMMs) has sparked a surge in research aimed at harnessing their remarkable reasoning abilities. However, for understanding text-rich images, challenges persist in fully leveraging the potential of LMMs, and existing methods struggle with effectively processing high-resolution images. In this work, we propose TextCoT, a novel Chain-of-Thought framework for text-rich image understanding. TextCoT utilizes the captioning ability of LMMs to grasp the global context of the image and the grounding capability to examine local textual regions. This allows for the extraction of both global and local visual information, facilitating more accurate question-answering. Technically, TextCoT consists of three stages, including image overview, coarse localization, and fine-grained observation. The image overview stage provides a comprehensive understanding of the global scene information, and the coarse localization stage approximates the image area containing the answer based on the question asked. Then, integrating the obtained global image descriptions, the final stage further examines specific regions to provide accurate answers. Our method is free of extra training, offering immediate plug-and-play functionality. Extensive experiments are conducted on a series of text-rich image question-answering benchmark datasets based on several advanced LMMs, and the results demonstrate the effectiveness and strong generalization ability of our method. Code is available at https://github.com/bzluan/TextCoT.

Figures (7)

• Figure 1: A pipeline comparison of (a) baseline LMM, (b) Zero-shot CoT zs-CoT, (c) CCoT cCoT, and (d) our proposed TextCoT. For enhanced comprehension of text-rich images, TextCoT leverages the captioning and grounding abilities of LMMs to grasp the global context and local textual regions within the image, respectively.
• Figure 2: An overview of the standard one-stage LMM (left) and our TextCoT (right). TextCoT comprises three stages: (1) Image Overview, (2) Coarse Localization, and (3) Fine-grained Observation. The initial two phases respectively generate a global context description $A_c$ of image $I_g$ and an answer region $A_g$ to question $Q$, facilitating the production of a more accurate response $A_f$.
• Figure 3: Comparison between the responses of LLaVA llava1.5 and those augmented by our TextCoT on Scene Text-Centric VQA datasets. The estimated answer regions $A_g$ in the second stage are highlighted in the image using yellow bounding boxes.
• Figure 4: Comparison between the responses of ShareGPT4V sharegpt4v and those augmented by our TextCoT on Scene Text-Centric VQA datasets. The estimated answer regions $A_g$ in the second stage are highlighted in the image using yellow bounding boxes.
• Figure 5: Comparison between the responses of LLaVA llava1.5 and those augmented by our TextCoT on Document-Oriented VQA datasets. The estimated answer regions $A_g$ in the second stage are highlighted in the image using red bounding boxes.