InteractiveOmni: A Unified Omni-modal Model for Audio-Visual Multi-turn Dialogue

TL;DR

InteractiveOmni presents a unified omni-modal large language model that processes image, video, audio, and text inputs to produce text and speech outputs in end-to-end fashion. By combining a vision encoder, an audio encoder, an LLM, and a streaming speech decoder, and by employing omni-modal pre-training followed by post-training with instruction tuning and DPO, it achieves strong multi-turn and memory capabilities. The work introduces new MMMB and MSIB benchmarks to robustly evaluate long-term memory and speech interaction, and demonstrates state-of-the-art or competitive performance across image, video, and audio understanding as well as speech generation for models of similar size. The open-source model shows notable efficiency, with the 4B variant rivaling larger counterparts on several benchmarks and the 8B variant excelling across tasks, underscoring practical impact for next-generation interactive AI systems.

Abstract

We introduce InteractiveOmni, a unified and open-source omni-modal large language model for audio-visual multi-turn interaction, ranging from 4B to 8B parameters, designed to lead the field of lightweight models by offering comprehensive omni-modal understanding and speech generation capabilities. To achieve this, we integrate the vision encoder, audio encoder, large language model, and speech decoder into a unified model for understanding and generation tasks. We design a multi-stage training strategy to ensure robust cross-modal capabilities, including pre-training for omni-modal understanding, followed by post-training with speech conversation and audio-visual interaction. To enable human-like long-term conversational ability, we meticulously curate a multi-turn training dataset that enhances the model's ability to handle complex and multi-turn interactions. To effectively evaluate the multi-turn memory and speech interaction capabilities, we construct the multi-modal multi-turn memory benchmark and the multi-turn speech interaction benchmark. Experiments demonstrate that InteractiveOmni significantly outperforms leading open-source models and provides a more intelligent multi-turn audio-visual experience, particularly in its long-term memory capabilities. Notably, InteractiveOmni-4B is comparable to the much larger model like Qwen2.5-Omni-7B on general benchmarks, and it can retain 97% of the performance of the InteractiveOmni-8B while utilizing only 50% of the model size. Achieving state-of-the-art results against similarly sized models across image, audio, video understanding, and speech generation tasks, InteractiveOmni is an accessible, open-source foundation for next-generation intelligent interactive systems.

Figures (7)

• Figure 1: Evaluation across image, video, and audio modalities on open-source benchmarks. InteractiveOmni outperforms the current leading multi-modal models such as Qwen2.5-VL-7Bbai2025qwen25, Kimi-Audio kimiteam2025kimiaudiotechnicalreport, MiniCPM-o-2.6 minicpm-v and Qwen2.5-Omni-7B xu2025qwen2.
• Figure 2: The schematic diagram of multi-turn audio-visual interaction. InteractiveOmni can perceive external audio and video inputs like a human, actively interact with users, and has the capabilities of multi-turn memory and empathy.
• Figure 3: The overview framework of InteractiveOmni. InteractiveOmni is composed of vision encoder, audio encoder, LLM decoder and streaming speech decoder. The extracted visual and audio tokens are processed by the LLM to generate text tokens and speech tokens sequentially.
• Figure 4: Data construction pipeline for multi-turn dialogue. In each turn, the visual element is sampled from a dedicated image and video repository. The corresponding question is then generated by a vision-language model using a specific prompt tailored to the desired question type. To ensure the dialogue effectively tests long-term memory, we specifically design turns that require recalling historical images and previous dialogue text. Finally, the generated text-format question and answer can be transformed into speech-based question-answer pairs using a TTS system, facilitating end-to-end training.
• Figure 5: The sketch of performance degradation with the increase of recall burden considering the turn distance and number of memorized images. InteractiveOmni is comparable to proprietary models like GPT-4o-mini and Gemini-2.5-Flash, consistently outperforming open-source models such as InternVL3-8B, Qwen2.5-VL-7B, and Qwen2.5-Omni-7B.