Be My Eyes: Extending Large Language Models to New Modalities Through Multi-Agent Collaboration

TL;DR

The paper tackles the unimodality limitation of large language models by introducing BeMyEyes, a modular multi-agent framework that pairs small, adaptable perceiver agents with powerful reasoner agents to enable multimodal reasoning without retraining large LLMs. A dedicated data synthesis and supervised fine-tuning pipeline trains the perceiver to communicate visual information effectively and align with the reasoner’s expectations, while keeping the reasoner frozen. Empirical results show BeMyEyes consistently improves multimodal reasoning across diverse benchmarks and model pairings, even outperforming some large proprietary VLMs on knowledge-intensive tasks and generalizing to specialized domains. The approach offers a scalable, open-source, and flexible alternative to full-scale multimodal model training with strong potential for extension to other modalities and future LLM advances.

Abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities in challenging, knowledge-intensive reasoning tasks. However, extending LLMs to perceive and reason over a new modality (e.g., vision), often requires costly development of large-scale vision language models (VLMs) with LLMs as backbones. Smaller VLMs are more efficient and adaptable but often lack the broad knowledge and reasoning capabilities of frontier LLMs. In this work, we propose BeMyEyes, a modular, multi-agent framework for extending LLMs to multimodal reasoning by orchestrating collaboration between efficient, adaptable VLMs as perceivers and powerful LLMs as reasoners through conversations. We then introduce a data synthesis and supervised fine-tuning pipeline to train the perceiver agent to effectively collaborate with the reasoner agent. By combining the complementary strengths of perception and reasoning agents, BeMyEyes avoids the need for training large-scale multimodal models, preserves the generalization and reasoning capabilities of LLMs, and allows flexible extension to new domains and modalities. Experiments show that our framework unlocks the multimodal reasoning capabilities for LLMs, enabling a lightweight and fully open-source solution, i.e. equipping text-only DeepSeek-R1 with Qwen2.5-VL-7B perceiver, to outperform large-scale proprietary VLMs such as GPT-4o on a wide range of knowledge-intensive multimodal tasks. These results demonstrate the effectiveness, modularity, and scalability of our multi-agent approach for building future multimodal reasoning systems.

Figures (3)

• Figure 1: Using BeMyEyesenables text-only models such as DeepSeek-R1 and GPT-4 to reach state-of-the-art performance on challenging multimodal benchmarks without modifying their parameters. Grey bars denote text-only baselines, where models receive only the benchmark questions without images. Dotted lines indicate GPT-4o performance.
• Figure 2: Overview of the BeMyEyesframework. A perceiver agent (small, adaptable VLM) extracts and summarizes visual information from input images, which is then communicated to the reasoner agent (large, frozen LLM) through multi-turn conversations. We also propose a data synthesis and supervised fine-tuning pipeline that allows us to train the perceiver agent to effectively collaborate with the reasoner agent. BeMyEyes's modular, multi-agent design decouples perception and reasoning, allowing text-only LLMs to perform multimodal reasoning without retraining, while enabling flexible integration of new perceiver or reasoner models.
• Figure 3: Error breakdown on MMMU Pro. We group all examples based on answer correctness under the single-perceiver, single-reasoner-with-vision, and BeMyEyes settings, respectively. The groups are represented by a three-light code signaling correct answers in each setting, and we order the groups based on their size. The bottom-left bar chart shows the total number of correctly answered examples for each setting.