Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Steer-MoE: Efficient Audio-Language Alignment with a Mixture-of-Experts Steering Module

Ruitao Feng, Bixi Zhang, Sheng Liang, Zheng Yuan

TL;DR

SteerMoE tackles the problem of efficiently aligning pretrained audio encoders with frozen LLM decoders by introducing a layer-wise Mixture-of-Experts steering module that operates entirely in the continuous embedding space. The approach treats audio representations as a dynamic, context-conditioned prompt for the LLM, enabling plug-and-play integration without modifying the LLM vocabulary or weights. Empirically, SteerMoE achieves competitive ASR results on LibriSpeech and AISHELL-2 and strong Clotho-AQA performance, while also showcasing the preserved agentic capabilities of the frozen LLM through simple function-calling tests. The work highlights modularity and parameter efficiency as viable paths for building sophisticated audio-language agents, with further work suggested on robustness to noise and deeper analysis of the learned experts.

Abstract

Aligning pretrained audio encoders and Large Language Models (LLMs) offers a promising, parameter-efficient path to building powerful multimodal agents. However, existing methods often require costly full-model finetuning or rely on static adapters that may lack expressive power. Drawing inspiration from the Platonic Representation Hypothesis, we introduce SteerMoE, a novel and modular framework for audio-language alignment. SteerMoE freezes both the audio encoder and the LLM decoder, training only a lightweight steering module integrated within the encoder's layers. This module uses a Mixture-of-Experts (MoE) router to dynamically select and apply learned steering vectors, progressively transforming continuous audio representations into a space comprehensible to the LLM. By operating entirely in the continuous embedding space, our approach requires no modifications to the LLM's vocabulary and preserves its advanced reasoning and agentic capabilities. We demonstrate through experiments on ASR, audio understanding, and a qualitative function-calling task that SteerMoE achieves strong performance while remaining highly modular and computationally efficient, offering a robust new paradigm for developing sophisticated audio-language systems.

Steer-MoE: Efficient Audio-Language Alignment with a Mixture-of-Experts Steering Module

TL;DR

SteerMoE tackles the problem of efficiently aligning pretrained audio encoders with frozen LLM decoders by introducing a layer-wise Mixture-of-Experts steering module that operates entirely in the continuous embedding space. The approach treats audio representations as a dynamic, context-conditioned prompt for the LLM, enabling plug-and-play integration without modifying the LLM vocabulary or weights. Empirically, SteerMoE achieves competitive ASR results on LibriSpeech and AISHELL-2 and strong Clotho-AQA performance, while also showcasing the preserved agentic capabilities of the frozen LLM through simple function-calling tests. The work highlights modularity and parameter efficiency as viable paths for building sophisticated audio-language agents, with further work suggested on robustness to noise and deeper analysis of the learned experts.

Abstract

Aligning pretrained audio encoders and Large Language Models (LLMs) offers a promising, parameter-efficient path to building powerful multimodal agents. However, existing methods often require costly full-model finetuning or rely on static adapters that may lack expressive power. Drawing inspiration from the Platonic Representation Hypothesis, we introduce SteerMoE, a novel and modular framework for audio-language alignment. SteerMoE freezes both the audio encoder and the LLM decoder, training only a lightweight steering module integrated within the encoder's layers. This module uses a Mixture-of-Experts (MoE) router to dynamically select and apply learned steering vectors, progressively transforming continuous audio representations into a space comprehensible to the LLM. By operating entirely in the continuous embedding space, our approach requires no modifications to the LLM's vocabulary and preserves its advanced reasoning and agentic capabilities. We demonstrate through experiments on ASR, audio understanding, and a qualitative function-calling task that SteerMoE achieves strong performance while remaining highly modular and computationally efficient, offering a robust new paradigm for developing sophisticated audio-language systems.
Paper Structure (15 sections, 5 equations, 1 figure, 3 tables)

This paper contains 15 sections, 5 equations, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Efficient Layer-wise Steering Module
  4. Modality Alignment via Continuous Prompting
  5. Training Objective
  6. Experiments
  7. Datasets and Tasks
  8. Implementation Details
  9. Main Results
  10. ASR Performance
  11. Audio Understanding Performance
  12. Qualitative Analysis of Agentic Capabilities
  13. Ablation Studies
  14. Discussion
  15. Compliance with Ethical Standards

Figures (1)

  • Figure 1: The SteerMoE architecture. A frozen audio encoder processes the input waveform. At each layer, a trainable MoE steering module refines the audio representations. The final steered features are projected and used as a continuous prompt for a frozen LLM decoder.