Omni-Captioner: Data Pipeline, Models, and Benchmark for Omni Detailed Perception

TL;DR

The paper tackles the challenge of fine-grained omni-modal perception by introducing a triad: Omni-Detective for agentic data generation, Omni-Captioner for high-detail audio–visual captions via a two-stage curriculum, and Omni-Cloze for stable, cross-modal evaluation. The two-stage training decouples audio grounding from joint audio–visual fusion, achieving strong performance on existing benchmarks and favorable detail–hallucination trade-offs. To address evaluation bottlenecks, Omni-Cloze provides a unified, cloze-style benchmark across audio, visual, and AV settings, with Not-Given options to differentiate omissions from hallucinations, showing high correlation with human judgments. Overall, the framework advances data-generation, modeling, and robust evaluation for fine-grained omni-modal perception, with potential impact on assistive AI, video understanding, and multi-modal reasoning.

Abstract

Fine-grained perception of multimodal information is critical for advancing human-AI interaction. With recent progress in audio-visual technologies, Omni Language Models (OLMs), capable of processing audio and video signals in parallel, have emerged as a promising paradigm for achieving richer understanding and reasoning. However, their capacity to capture and describe fine-grained details remains limited explored. In this work, we present a systematic and comprehensive investigation of omni detailed perception from the perspectives of the data pipeline, models, and benchmark. We first identify an inherent "co-growth" between detail and hallucination in current OLMs. To address this, we propose Omni-Detective, an agentic data generation pipeline integrating tool-calling, to autonomously produce highly detailed yet minimally hallucinatory multimodal data. Based on the data generated with Omni-Detective, we train two captioning models: Audio-Captioner for audio-only detailed perception, and Omni-Captioner for audio-visual detailed perception. Under the cascade evaluation protocol, Audio-Captioner achieves the best performance on MMAU and MMAR among all open-source models, surpassing Gemini 2.5 Flash and delivering performance comparable to Gemini 2.5 Pro. On existing detailed captioning benchmarks, Omni-Captioner sets a new state-of-the-art on VDC and achieves the best trade-off between detail and hallucination on the video-SALMONN 2 testset. Given the absence of a dedicated benchmark for omni detailed perception, we design Omni-Cloze, a novel cloze-style evaluation for detailed audio, visual, and audio-visual captioning that ensures stable, efficient, and reliable assessment. Experimental results and analysis demonstrate the effectiveness of Omni-Detective in generating high-quality detailed captions, as well as the superiority of Omni-Cloze in evaluating such detailed captions.

Figures (12)

• Figure 1: Comparison of the detailed captioning among the Omni-Captioner and other omni models.
• Figure 2: Relationship between caption length, detail coverage, and hallucination on Gemini‑2.5‑Pro in the detailed captioning task.
• Figure 3: Omni-Detective. An agentic data generation and cleaning pipeline integrating specialist tools for omni detailed perception.
• Figure 4: Omni-Cloze utilizes cloze-style MCQ to evaluate models' detailed captioning abilities.
• Figure 5: Data curation pipeline for Omni-Cloze.