OmniGAIA: Towards Native Omni-Modal AI Agents

TL;DR

This work introduces OmniGAIA, a comprehensive benchmark designed to evaluate omni-modal agents on tasks necessitating deep reasoning and multi-turn tool execution across video, audio, and image modalities, and proposes OmniAtlas, a native omni-modal foundation agent under tool-integrated reasoning paradigm with active omni-modal perception.

Abstract

Human intelligence naturally intertwines omni-modal perception -- spanning vision, audio, and language -- with complex reasoning and tool usage to interact with the world. However, current multi-modal LLMs are primarily confined to bi-modal interactions (e.g., vision-language), lacking the unified cognitive capabilities required for general AI assistants. To bridge this gap, we introduce OmniGAIA, a comprehensive benchmark designed to evaluate omni-modal agents on tasks necessitating deep reasoning and multi-turn tool execution across video, audio, and image modalities. Constructed via a novel omni-modal event graph approach, OmniGAIA synthesizes complex, multi-hop queries derived from real-world data that require cross-modal reasoning and external tool integration. Furthermore, we propose OmniAtlas, a native omni-modal foundation agent under tool-integrated reasoning paradigm with active omni-modal perception. Trained on trajectories synthesized via a hindsight-guided tree exploration strategy and OmniDPO for fine-grained error correction, OmniAtlas effectively enhances the tool-use capabilities of existing open-source models. This work marks a step towards next-generation native omni-modal AI assistants for real-world scenarios.

Figures (6)

• Figure 1: Short examples from OmniGAIA. Two illustrative (image + audio and video w/ audio) instances showing omni-modal evidence integration and multi-step tool use (e.g., web search) to derive a verifiable final answer.
• Figure 2: OmniGAIA construction pipeline. From video w/ audio and image + audio data, we mine key signals, build and expand a tool-augmented event graph, and generate LLM and human-verified multi-hop QA via event fuzzification.
• Figure 3: OmniGAIA statistics. This figure presents a detailed breakdown of domain distributions, required capabilities, and task attributes, underscoring the complex demands placed on omni-modal perception, reasoning, and tool utilization.
• Figure 4: OmniAtlas training strategy. Left, we synthesize tool-integrated trajectories via step-level supervision and guided tree exploration, selecting successful runs for supervised fine-tuning; right, OmniDPO locates the first error in a failed trajectory and generates a corrected prefix, forming positive/negative preference pairs for fine-grained correction.
• Figure 5: Fine-grained error analysis. These heatmaps illustrate the frequency of specific error types—including failures in instruction following, visual/audio perception, tool usage, reasoning, and absence of an answer—across six different models.