DeepOmni: Towards Seamless and Smart Speech Interaction with Adaptive Modality-Specific MoE

TL;DR

DeepOmni tackles catastrophic forgetting in native multimodal LLMs by introducing an adaptive modality-specific mixture-of-experts that isolates speech and text knowledge within a shared backbone. It employs a three-stage training protocol plus cross-modal joint training, augmented with DPO-based reinforcement learning for stable audio generation, to maintain language capabilities while delivering end-to-end speech interaction with latency under 0.5 seconds. Empirically, it achieves a 5.5% relative loss in language ability—comparable to modular MLLMs—while outperforming other native MLLMs on speech-related tasks and preserving LLM performance, demonstrating a practical path toward robust, low-latency native multimodal systems. The approach bridges native and modular paradigms and provides a scalable framework for mitigating catastrophic forgetting in multimodal LLMs, with code and models released for reproducibility.

Abstract

Native multimodal large language models (MLLMs) restructure a single large language model (LLM) into a spoken language model (SLM) capable of both speech and text generation. Compared to modular and aligned MLLMs, native MLLMs preserve richer paralinguistic features such as emotion and prosody, and generate speech responses directly within the backbone LLM rather than using a separate speech decoder. This integration also results in lower response latency and smoother interaction. However, native MLLMs suffer from catastrophic forgetting and performance degradation because the available paired speech-text data is insufficient to support the pretraining of MLLMs compared to the vast amount of text data required to pretrain text LLMs. To address this issue, we propose DeepTalk, a framework for adaptive modality expert learning based on a Mixture of Experts (MoE) architecture. DeepTalk first adaptively distinguishes modality experts according to their modality load within the LLM. Each modality expert then undergoes specialized single-modality training, followed by joint multimodal collaborative training. As a result, DeepTalk incurs only a 5.5% performance drop compared to the original LLM, which is significantly lower than the average performance drop of over 20% typically seen in native MLLMs (such as GLM-4-Voice), and is on par with modular MLLMs. Meanwhile, the end-to-end dialogue latency remains within 0.5 seconds, ensuring a seamless and intelligent speech interaction experience. Code and models are released at https://github.com/talkking/DeepTalk.

Figures (6)

• Figure 1: Comparison of DeepOmni and existing voice interaction systems. Compared to modular MLLMs, DeepOmni embeds acoustic experts into a single pre-trained LLM and integrates both speech and language decoding within a single LLM. Through adaptive modality expert selection and a three-stage training process for modality experts, DeepOmni effectively alleviates catastrophic forgetting in text-based LLMs while maintaining robust voice interaction capabilities.
• Figure 2: The DeepOmni model architecture.
• Figure 3: The training process consists of three stages: stage 1 is for modality alignment, stage 2 is for training single modality experts, and stage 3 is for joint training of the modality experts.
• Figure 4: The modality load of the experts in the first layers.
• Figure 5: The modality load of the experts in the middle layers.