Agent-Omni: Test-Time Multimodal Reasoning via Model Coordination for Understanding Anything

TL;DR

Agent-Omni introduces a training-free framework that coordinates specialized foundation models at test time to achieve omni-modal reasoning over text, image, audio, and video. It uses a master-agent loop with stages for Perception, Reasoning, Execution, and Decision, plus a modular Model Pool to route tasks to modality-specific models and iteratively refine answers. Across text, image, video, audio, and omni benchmarks, Agent-Omni achieves competitive or state-of-the-art accuracy, particularly on challenging cross-modal and omni tasks, albeit with higher latency due to coordination. The approach preserves the strengths of individual experts, avoids costly joint training, and offers a transparent, extensible solution with open-source potential for future enhancements.

Abstract

Multimodal large language models (MLLMs) have shown strong capabilities but remain limited to fixed modality pairs and require costly fine-tuning with large aligned datasets. Building fully omni-capable models that can integrate text, images, audio, and video remains impractical and lacks robust reasoning support. In this paper, we propose an Agent-Omni framework that coordinates existing foundation models through a master-agent system, enabling flexible multimodal reasoning without retraining. The master agent interprets user intent, delegates subtasks to modality-specific agents, and integrates their outputs into coherent responses. Extensive experiments across text, image, audio, video, and omni benchmarks show that Agent-Omni consistently achieves state-of-the-art performance, particularly on tasks requiring complex cross-modal reasoning. Its agent-based design enables seamless integration of specialized foundation models, ensuring adaptability to diverse inputs while maintaining transparency and interpretability. In addition, the framework is modular and easily extensible, allowing future improvements as stronger models become available.

Figures (7)

• Figure 1: Comparison of Agent-Omni and other omni methods across multimodal benchmarks.
• Figure 2: Overview of the Agent-Omni framework. A master agent interprets the query, identifies relevant modalities, and delegates sub-questions to corresponding foundation models (text, image, audio, video). Their outputs are iteratively integrated and refined through a self-improvement loop, enabling coherent multimodal reasoning in test-time inference.
• Figure 3: The prompt template used in experiments.
• Figure 4: The prompt template of reasoning stage.
• Figure 5: The prompt template of decision stage.