OmniAgent: Audio-Guided Active Perception Agent for Omnimodal Audio-Video Understanding

TL;DR

OmniAgent addresses the challenge of fine-grained cross-modal understanding in omnimodal systems by introducing an audio-guided active perception agent that dynamically orchestrates modality-specific tools through a Think-Act-Observe-Reflect loop. By leveraging audio cues for coarse localization and an audio-guided event localization mechanism, the framework enables flexible, coarse-to-fine reasoning across audio and video streams. It defines a modality-aware toolkit with video, audio, and event tools, and demonstrates state-of-the-art results across Daily-Omni, OmniVideoBench, and WorldSense benchmarks, with improvements of 10–20 percentage points over strong baselines. The work highlights the potential of active perception to overcome static alignment bottlenecks and suggests future extensions to additional modalities and end-to-end omnimodal agents with memory caching.

Abstract

Omnimodal large language models have made significant strides in unifying audio and visual modalities; however, they often lack the fine-grained cross-modal understanding and have difficulty with multimodal alignment. To address these limitations, we introduce OmniAgent, a fully audio-guided active perception agent that dynamically orchestrates specialized tools to achieve more fine-grained audio-visual reasoning. Unlike previous works that rely on rigid, static workflows and dense frame-captioning, this paper demonstrates a paradigm shift from passive response generation to active multimodal inquiry. OmniAgent employs dynamic planning to autonomously orchestrate tool invocation on demand, strategically concentrating perceptual attention on task-relevant cues. Central to our approach is a novel coarse-to-fine audio-guided perception paradigm, which leverages audio cues to localize temporal events and guide subsequent reasoning. Extensive empirical evaluations on three audio-video understanding benchmarks demonstrate that OmniAgent achieves state-of-the-art performance, surpassing leading open-source and proprietary models by substantial margins of 10% - 20% accuracy.

Figures (5)

• Figure 1: (a): Illustration of OmniAgent, an audio-guided active perception agent designed for omnimodal understanding. Given a complex user query, our agent employs a recursive "Think-Act-Observe-Reflect" loop and actively orchestrates multimodal tools (video, audio, and event tools) for fine-grained audio-video understanding. The presented video clip is from a real-life vlog; the question is about two Chinese characters on a hanging signboard in the video. Initially, the agent utilizes acoustic cues to locate the temporal segment containing the key information ("the kitten"), then invokes the video clip tool within that time window. Within the salient segment, given the same GPU memory budget, we can afford model inference at an increased spatial and temporal resolution. With sufficient relevant visual evidence and the audio as input, the agent derives the correct answer. In contrast, the end-to-end video understanding model Qwen3-Omni xu2025qwen3(b) cannot achieve such a fine-grained understanding and gives the wrong answer. (c): Performance comparison on three audio-video understanding benchmarks. OmniAgent demonstrates superior performance, consistently outperforming strong end-to-end OmniLLMs such as Qwen3-Omni xu2025qwen3 and Gemini 2.5-Flash comanici2025gemini.
• Figure 2: (a) End-to-end OmniLLMs implicitly fuse modalities but suffer from high training costs, difficult alignment, and limited fine-grained reasoning. (b) Fixed workflow agents rely on rigid pipelines, lacking the flexibility to allocate attention for fine-grained analysis adaptively. (c) Caption-based agents incur high precomputation costs and noise sensitivity, often failing to capture comprehensive multimodal context. (d) Our OmniAgent employs active perception reasoning and inquiry. Within an iterative reflective loop, the agent strategically calls on the ability of video and audio understanding. This explicitly solves the cross-modal alignment difficulty and achieves fine-grained understanding.
• Figure 3: Overview of the OmniAgent framework. The system processes audio and video inputs through an iterative thinking-action-observe-reflection cycle. The agent utilizes a comprehensive suite of perception tools (video, audio, and event) to gather fine-grained evidence, while the reflection module synthesizes observations to update the memory and decide whether to rethink or conclude the task.
• Figure 4: Visualization of the responses and underlying reasoning processes generated by our OmniAgent and Gemini2.5-Flash to an audio-video understanding question.
• Figure 5: Analysis of the behavior of OmniAgent with different core LLM models. We quantified tool utilization patterns by calculating both the proportion of invocations (call ratio) and the average number of reasoning steps per call. In the resulting visualization, the sector angle represents the tool call ratio, and the magnitude of the radius denotes the specific execution steps at which the tool was invoked.