UniMoE-Audio: Unified Speech and Music Generation with Dynamic-Capacity MoE

TL;DR

UniMoE-Audio tackles the challenge of unified speech and music generation by introducing a Dynamic-Capacity Mixture-of-Experts (MoE) with Top-P routing and a hybrid expert design that includes routed, shared, and null experts. A data-aware, three-stage training curriculum—Independent Specialist Training, MoE Integration and Warmup, and Synergistic Joint Training—addresses the persistent data imbalance and task conflict between speech and music tasks. Empirically, the model achieves state-of-the-art or competitive results across major speech and music benchmarks, while analyses of routing behavior reveal specialized expert usage and adaptive computation patterns that mitigate interference between domains. The work demonstrates that specialized MoE architectures, combined with carefully staged training, offer a scalable path toward robust, synergistic universal audio generation with practical implications for multimodal AI systems.

Abstract

Recent advances in unified multimodal models indicate a clear trend towards comprehensive content generation. However, the auditory domain remains a significant challenge, with music and speech often developed in isolation, hindering progress towards universal audio synthesis. This separation stems from inherent task conflicts and severe data imbalances, which impede the development of a truly unified audio generation model. To address this challenge, we propose UniMoE-Audio, a unified speech and music generation model within a novel Dynamic-Capacity Mixture-of-Experts (MoE) framework. Architecturally, UniMoE-Audio introduces a Top-P routing strategy for dynamic expert number allocation, and a hybrid expert design comprising routed experts for domain-specific knowledge, shared experts for domain-agnostic features, and null experts for adaptive computation skipping. To tackle data imbalance, we introduce a three-stage training curriculum: 1) Independent Specialist Training leverages original datasets to instill domain-specific knowledge into each "proto-expert" without interference; 2) MoE Integration and Warmup incorporates these specialists into the UniMoE-Audio architecture, warming up the gate module and shared expert using a subset of balanced dataset; and 3) Synergistic Joint Training trains the entire model end-to-end on the fully balanced dataset, fostering enhanced cross-domain synergy. Extensive experiments show that UniMoE-Audio not only achieves state-of-the-art performance on major speech and music generation benchmarks, but also demonstrates superior synergistic learning, mitigating the performance degradation typically seen in naive joint training. Our findings highlight the substantial potential of specialized MoE architecture and curated training strategies in advancing the field of universal audio generation. Homepage: https://mukioxun.github.io/Uni-MoE-site/home.html

Figures (5)

• Figure 1: Performance of UniMoE-Audio. Left: Comparison against specialized baselines reveals the failure of naive joint training, which causes a clear performance degradation on speech generation and more significant decline on music generation. In contrast, our UniMoE-Audio yields synergistic gains across both tasks. Right: Radar charts show UniMoE-Audio achieving the best comprehensive performance against leading models on a wide array of speech (a) and music (b) metrics.
• Figure 2: An overview of the UniMoE-Audio framework. Left: UniMoE-Audio is a unified model capable of performing speech and music generation by leveraging multimodal conditional inputs, including Voice Cloning, Text-to-Speech (TTS), Text-to-Music (T2M), and Video-to-Music (V2M). Center: The core architecture of our model is a Transformer with Dynamic-Capacity MoE layers. Right: We propose a novel Top-P routing strategy, which dynamically selects the number of experts allocated to each token based on their complexity.
• Figure 3: Training loss for the speech generation task (top) and music generation task (bottom). The plots show the transition from the Warmup Training Stage (blue) to the Synergistic Joint Training Stage (orange). The solid line represents the moving average of the loss.
• Figure 4: Visualization of the dynamic computational budget allocated by our Top-P routing mechanism. The figure illustrates the proportion of tokens activating a varying number of experts at each layer, revealing a "rise-and-fall" pattern where more computational resources are adaptively assigned to the middle layers. For clarity, counts of activated experts are grouped into bins (e.g., "Expert 2-3" represents tokens activating either 2 or 3 experts).
• Figure 5: Analysis of expert routing dynamics in UniMoE-Audio across transformer layers. The top-left "All Experts" plot illustrates the routing frequency for each of the eight routed experts (E1-E8, colored) and the null expert (E9, gray). The subsequent nine plots provide a granular breakdown for each expert, showing the proportion of tokens routed from the Music (lighter shade) versus the TTS (darker shade) task.