MoST: Mixing Speech and Text with Modality-Aware Mixture of Experts

TL;DR

MoST addresses the challenge of unifying speech and text within a single LLM by introducing Modality-Aware Mixture of Experts (MAMoE), which partitions experts into modality-specific groups and shared experts to enable specialized learning and cross-modal transfer. The model processes continuous audio with a HuBERT-based encoder and synthesizes speech via a HiFiGAN vocoder, while text is handled with standard tokenization, all feeding a MoE-based decoder that routes tokens according to modality indicators. A two-stage, data-efficient pipeline — cross-modal post-training on open ASR/TTS datasets followed by mixed speech-text instruction fine-tuning — enables strong multimodal capabilities while preserving speech strengths, and relies entirely on open-source data. Empirical results across ASR, TTS, audio language modeling, and spoken QA show MoST achieving state-of-the-art or competitive performance for models with comparable parameters, with ablations confirming the efficacy of modality-specific routing and shared-expert interactions. MoST stands out as a fully open-source speech-text LLM built on a Mixture of Experts architecture, offering a scalable path toward accessible, multimodal language models and highlighting future directions including additional modalities and larger-scale deployments.

Abstract

We present MoST (Mixture of Speech and Text), a novel multimodal large language model that seamlessly integrates speech and text processing through our proposed Modality-Aware Mixture of Experts (MAMoE) architecture. While current multimodal models typically process diverse modality representations with identical parameters, disregarding their inherent representational differences, we introduce specialized routing pathways that direct tokens to modality-appropriate experts based on input type. MAMoE simultaneously enhances modality-specific learning and cross-modal understanding through two complementary components: modality-specific expert groups that capture domain-specific patterns and shared experts that facilitate information transfer between modalities. Building on this architecture, we develop an efficient transformation pipeline that adapts the pretrained MoE language model through strategic post-training on ASR and TTS datasets, followed by fine-tuning with a carefully curated speech-text instruction dataset. A key feature of this pipeline is that it relies exclusively on fully accessible, open-source datasets to achieve strong performance and data efficiency. Comprehensive evaluations across ASR, TTS, audio language modeling, and spoken question answering benchmarks show that MoST consistently outperforms existing models of comparable parameter counts. Our ablation studies confirm that the modality-specific routing mechanism and shared experts design significantly contribute to performance gains across all tested domains. To our knowledge, MoST represents the first fully open-source speech-text LLM built on a Mixture of Experts architecture. \footnote{We release MoST model, training code, inference code, and training data at https://github.com/NUS-HPC-AI-Lab/MoST

Figures (6)

• Figure 1: MoST overall architecture. MoST supports interleaved text and audio as input and can generate both text and audio. MoST blocks are transformer blocks featured with Modality-Aware Mixture of Experts (MAMoE). MAMoE is comprised of three critical components: modality-specific expert groups, cross-modal shared expert, and the modality-aware router. Modality-aware router utilizes the modality indicator information for modality-specific routing.
• Figure 2: MoST training recipe overview. The left panel illustrates the systematic transformation pipeline from MoE LLM to MoST, encompassing data preparation and two-stage training protocol. The right panel details the interrupted dialogue synthesis process for instruction dataset construction.
• Figure 3: Performance comparison on Spoken Question Answering tasks. MoST achieves competitive or superior performances on all 3 datasets.
• Figure 4: Controlled comparison with identical Llama3.2 3B initialization. We report (100%-WER) for ASR and TTS tasks and accuracy for other tasks.
• Figure 5: MAMoE design variants analysis. The 2×3 grid displays: Training loss curves; Validation loss curves; ASR, TTS performance (WER) on LibriSpeech dev-clean; Audio language modeling performance (Accuracy) on sWUGGY; Spoken QA performance (Accuracy) on LlamaQ. All metrics are reported at 10,000 training steps. Full MAMoE consistently outperforms ablated versions.