Discovering and Causally Validating Emotion-Sensitive Neurons in Large Audio-Language Models

TL;DR

This work demonstrates, with causal validation, that emotion-sensitive neurons (ESNs) exist in three open-source large audio-language models and can be identified via emotion-conditioned activation statistics. Through inference-time deactivation and gain-based steering, ESNs show emotion-specific self-effects and controllable bias toward target emotions, with MAD and CAS selectors yielding the strongest causality signals. ESNs exhibit non-uniform layer locality and partial cross-dataset transfer, and agnostic injection strategies reveal complex interactions among emotion pathways. The findings provide a neuron-level account of affective decisions in LALMs and suggest actionable handles for interpretable and controllable affective behavior in speech-enabled systems.

Abstract

Emotion is a central dimension of spoken communication, yet, we still lack a mechanistic account of how modern large audio-language models (LALMs) encode it internally. We present the first neuron-level interpretability study of emotion-sensitive neurons (ESNs) in LALMs and provide causal evidence that such units exist in Qwen2.5-Omni, Kimi-Audio, and Audio Flamingo 3. Across these three widely used open-source models, we compare frequency-, entropy-, magnitude-, and contrast-based neuron selectors on multiple emotion recognition benchmarks. Using inference-time interventions, we reveal a consistent emotion-specific signature: ablating neurons selected for a given emotion disproportionately degrades recognition of that emotion while largely preserving other classes, whereas gain-based amplification steers predictions toward the target emotion. These effects arise with modest identification data and scale systematically with intervention strength. We further observe that ESNs exhibit non-uniform layer-wise clustering with partial cross-dataset transfer. Taken together, our results offer a causal, neuron-level account of emotion decisions in LALMs and highlight targeted neuron interventions as an actionable handle for controllable affective behaviors.

Figures (6)

• Figure 1: Per-emotion accuracy-change heatmaps for Qwen2.5-Omni-7B under neuron ablation, reported on IEMOCAP (top), MELD (middle), and MSP-Podcast (bottom). Rows index the source emotion used to identify the ESN mask; columns index the evaluation emotion subset. All values are absolute accuracy differences with respect to the unintervened model. Diagonal entries correspond to self-effects, while off-diagonal cells reflect cross-effects.
• Figure 2: Sensitivity to intervention budget and identification-pool size. (a--c) Accuracy-change heatmaps as we vary the deactivated fraction $r$ of ESNs. (d) Accuracies as we vary the number of correctly answered identification examples per emotion used to construct the ESN masks (Qwen2.5-Omni-7B, CAS-selected ESNs, MSP-Podcast).
• Figure 3: Accuracy-change $\Delta$ heatmaps on MELD for different steering strengths$\alpha$ (CAS, Qwen2.5-Omni-7B).
• Figure 4: Layer-wise distribution of identified ESNs by MAD (subfigure a, b, c) and CAS (subfigure d, e, f). All three models have 28-layer decoders. The color is log-scaled for better readability.
• Figure 5: Accuracy-change heatmaps on cross-dataset deactivation. The results of Qwen2.5-Omni-7B using CAS selector are shown. Note that while all datasets contain "anger", "happiness/joy", "neutral" and "sadness"; MELD and MSP-Podcast additionally share "surprise". Appendix \ref{['appendix:cross_results']} presents the remaining two directions.