SEE: Signal Embedding Energy for Quantifying Noise Interference in Large Audio Language Models

TL;DR

This work tackles the robustness of Large Audio Language Models (LALMs) to real-world noise by introducing Signal Embedding Energy (SEE), a metric that quantifies semantic interference within the embedding space. SEE is computed from structured activation subspaces derived from aligned clean and noisy inputs, and it correlates strongly with generation quality ($r=0.98$). Traditional waveform denoising often fails to reduce SEE and can even worsen it, revealing a mismatch between acoustic fidelity and semantic reasoning. To address this, the authors propose Signal Embedding Energy Neutralization (SEEN), a training-free method that suppresses noise-aligned components in embedding activations, yielding about a 6.7% average improvement in generation performance across noise types and models, and demonstrating transferability across unseen noise conditions.

Abstract

Large Audio Language Models (LALMs) have been widely applied in real-time scenarios, such as in-car assistants and online meeting comprehension. In practice, audio inputs are often corrupted by device and environmental noise, leading to performance degradation. However, existing LALM studies on noise lack quantitative analysis and rely mainly on intuition and empirical observation, thus failing to understand practical robustness. To address this issue, we introduce Signal Embedding Energy (SEE), a method for quantifying the impact of noise intensity on LALM inputs, enabling the differentiation of LALM robustness in real-world deployments. SEE introduces a perspective based on structured activation subspaces derived from the model's internal representations, which more accurately captures its perception of noise than raw audio features. Across experiments, SEE exhibits a strong correlation with LALM performance, achieving a correlation of 0.98. Surprisingly, traditional audio denoising methods are only marginally effective for LALMs, and, in some cases, even increase SEE and impair performance. This suggests a mismatch between speech-centric denoising objectives and the noise sensitivity of modern LALMs. Therefore, we propose a mitigation strategy derived from SEE to denoise LALM inputs, outperforming existing denoising methods. This paper introduces a novel metric for noise quantification in LALMs, providing guidance for robustness improvements in real-world deployments.

Figures (7)

• Figure 1: Motivation and overview of representation-level noise robustness in LALMs. Waveform-level denoising improves acoustic quality but may introduce semantic interference, which is quantified by Signal Embedding Energy (SEE) and mitigated by SEEN.
• Figure 2: Offline, we construct noise activation matrices, localize noise dominant layers, and extract principal noise directions to form a noise subspace. Online, SEE quantifies activation energy projected onto this subspace, and SEEN mitigates noise by removing the projected components at the embedding layers.
• Figure 3: SEE is negatively correlated with GSR, with a consistent trend observed across different task types.
• Figure 4: Generation success rate (GSR) versus SNR, showing a sharp degradation below $\text{SNR}=10$.
• Figure 5: SEEN improves generation quality across noise types without consistent performance degradation.