Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SeeingEye: Agentic Information Flow Unlocks Multimodal Reasoning In Text-only LLMs

Weijia Zhang, Zijia Liu, Haoru Li, Haoqi Chen, Jiaxuan You

TL;DR

SeeingEye introduces a two-agent framework that decouples perception from reasoning to enable multimodal reasoning in text-only LLMs via a Structured Intermediate Representation (SIR). The Translator Agent converts visual input into a rich, query-relevant SIR through adaptive tool use and Visual Chain-of-Thought, while the Reasoning Agent performs high-level cognition on the SIR with multi-round feedback. Experiments on MMMU, MMMU-Pro, OCR-BenchV2, and MIA-Bench show the approach surpasses larger end-to-end VLMs and offers cost efficiency through modular design. The work demonstrates that agentic information flow and structured communication between specialized agents provide a scalable path to leverage strong text-only LLMs for complex multimodal reasoning tasks.

Abstract

Recent advances in text-only large language models (LLMs), such as DeepSeek-R1, demonstrate remarkable reasoning ability. However, these models remain fragile or entirely incapable when extended to multi-modal tasks. Existing approaches largely rely on single-form captions, which lack diversity and often fail to adapt across different types of Visual Question Answering (VQA) benchmarks. As a result, they provide no principled or efficient channel for transmitting fine-grained visual information. We introduce Seeing Eye, a modular framework that unlocks multimodal reasoning in text-only LLMs through an agent-based small VLM translator. This translator acts as a perception agent: it can invoke specialized tools (e.g., OCR and crop) and iteratively distill multimodal inputs into structured intermediate representations (SIRs) tailored to the question. These SIRs are then passed to the text-only LLM, which serves as a reasoning agent. Crucially, the translator and reasoner engage in multi-round feedback and interaction, enabling the extraction of targeted visual details and yielding more confident answers. Experiments on knowledge-intensive VQA benchmarks, including MMMU and MIA-Bench, demonstrate that Seeing Eye not only reduces inference cost but also surpasses much larger end-to-end VLMs. For example, an instantiation combining a 3B-parameter vision translator with an 8B-parameter language reasoner outperforms a monolithic 32B VLM on challenging knowledge-based questions. Our results highlight that decoupling perception from reasoning via agent information flow offers a scalable and plug-and-play pathway to multimodal reasoning, allowing strong text-only LLMs to fully leverage their reasoning capabilities. Code is available at: https://github.com/ulab-uiuc/SeeingEye

SeeingEye: Agentic Information Flow Unlocks Multimodal Reasoning In Text-only LLMs

TL;DR

SeeingEye introduces a two-agent framework that decouples perception from reasoning to enable multimodal reasoning in text-only LLMs via a Structured Intermediate Representation (SIR). The Translator Agent converts visual input into a rich, query-relevant SIR through adaptive tool use and Visual Chain-of-Thought, while the Reasoning Agent performs high-level cognition on the SIR with multi-round feedback. Experiments on MMMU, MMMU-Pro, OCR-BenchV2, and MIA-Bench show the approach surpasses larger end-to-end VLMs and offers cost efficiency through modular design. The work demonstrates that agentic information flow and structured communication between specialized agents provide a scalable path to leverage strong text-only LLMs for complex multimodal reasoning tasks.

Abstract

Recent advances in text-only large language models (LLMs), such as DeepSeek-R1, demonstrate remarkable reasoning ability. However, these models remain fragile or entirely incapable when extended to multi-modal tasks. Existing approaches largely rely on single-form captions, which lack diversity and often fail to adapt across different types of Visual Question Answering (VQA) benchmarks. As a result, they provide no principled or efficient channel for transmitting fine-grained visual information. We introduce Seeing Eye, a modular framework that unlocks multimodal reasoning in text-only LLMs through an agent-based small VLM translator. This translator acts as a perception agent: it can invoke specialized tools (e.g., OCR and crop) and iteratively distill multimodal inputs into structured intermediate representations (SIRs) tailored to the question. These SIRs are then passed to the text-only LLM, which serves as a reasoning agent. Crucially, the translator and reasoner engage in multi-round feedback and interaction, enabling the extraction of targeted visual details and yielding more confident answers. Experiments on knowledge-intensive VQA benchmarks, including MMMU and MIA-Bench, demonstrate that Seeing Eye not only reduces inference cost but also surpasses much larger end-to-end VLMs. For example, an instantiation combining a 3B-parameter vision translator with an 8B-parameter language reasoner outperforms a monolithic 32B VLM on challenging knowledge-based questions. Our results highlight that decoupling perception from reasoning via agent information flow offers a scalable and plug-and-play pathway to multimodal reasoning, allowing strong text-only LLMs to fully leverage their reasoning capabilities. Code is available at: https://github.com/ulab-uiuc/SeeingEye

Paper Structure

This paper contains 26 sections, 6 equations, 14 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. The Translator Agent: Grounded Visual Analysis
  5. Visual Chain-of-Thought (VCoT) Analysis.
  6. Adaptive Tool Selection and Execution.
  7. Iterative SIR Refinement and Termination.
  8. The Reasoning Agent: High-Level Cognition and Decision-Making
  9. SIR-Grounded Analysis and Tool Use.
  10. Terminal Decision Policy.
  11. Experiments
  12. Experimental Settings
  13. Main Results
  14. Superiority over End-to-End VLMs.
  15. Effectiveness of the Agentic Information Flow.
  16. ...and 11 more sections

Figures (14)

  • Figure 1: The Agentic Information Flow of our SeeingEye framework. The process begins with the Translator Agent (left), which takes the previous state's SIR and external feedback to perform a Visual Chain-of-Thought (VCoT) analysis. It uses tools to gather new visual evidence, reflects on the results, and iteratively refines the SIR. If the information is deemed sufficient (PASS), the improved SIR is passed to the Text Reasoning Agent (right). The Reasoner performs its own CoT-driven analysis and tool use. Based on its confidence, it either produces a final answer (PASS) or generates targeted feedback (FAIL), initiating a new outer loop iteration for the Translator to gather more specific visual cues.
  • Figure 2: A detailed case study of the SmartGridCaption tool. (1) An initial global SIR fails to identify the animal in the poster. (2) The tool grids the image and locates the relevant patch [9, 9]. (3) A fine-grained patch caption is generated and integrated, updating the SIR with the crucial detail of a "dove". (4) This refined SIR enables the Reasoning Agent to provide a high-confidence final answer in a single iteration.
  • Figure 3: A Case Study of SIR.
  • Figure 4: SIR JSON Structure. The global_caption and confidence fields are provided by the Translator Agent (via the TerminateAndOutputCaption tool). The feedback field is provided by the Text Reasoning Agent (via the TERMINATE_AND_ASK_TRANSLATOR tool).
  • Figure 5: Translator Agent SIR Management Prompt. This defines how the Translator iteratively builds the SIR and the output format it must adhere to.
  • ...and 9 more figures