Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Adaptive vector steering: A training-free, layer-wise intervention for hallucination mitigation in large audio and multimodal models

Tsung-En Lin, Kuan-Yi Lee, Hung-Yi Lee

TL;DR

The paper addresses hallucinations in large audio-language models by introducing Adaptive Vector Steering (AVS), a training-free, inference-time intervention that grounds model outputs in audio content. AVS rests on a layer-wise influence analysis showing later layers disproportionately shape final decisions, and it uses a contrastive steering vector to push activations toward audio grounding, with an adaptive scheme that strengthens later layers while preserving total steering strength. Empirical results on two diverse models (Qwen2-Audio-7B-Instruct and Gemma-3n-E4B-It) across Audio Hallucination QA and MMAU benchmarks demonstrate consistent improvements in F1 and accuracy, notably raising Gemma’s F1 to 0.619 and Qwen’s MMAU accuracy to 0.592 in the Test set. The work contributes a practical, interpretable, and generalizable mechanism for reducing ungrounded content in multimodalAudio-Language Models without fine-tuning, with potential implications for reliability and deployment of audio-grounded AI systems.

Abstract

Large Audio-Language Models and Multi-Modal Large Language Models have demonstrated strong capabilities in tasks such as Audio Question Answering (AQA), Audio Captioning, and Automatic Speech Recognition (ASR). However, there is growing evidence that these models can hallucinate about the content of the audio. To address this issue, we probe the models' internal states and propose Adaptive Vector Steering (AVS), a method that better grounds generation in audio content. We also identify a strong correlation between output correctness and internal representations. Experiments show consistent performance gains across two models and two benchmarks. On the Audio Hallucination QA dataset, our method boosts the F1-score of Gemma from 0.550 to 0.619 and Qwen from 0.626 to 0.632. Furthermore, our method increases the accuracy of Qwen on MMAU from 0.548 to 0.592, marking an 8% relative increase. To the best of our knowledge, this is the first work to apply vector steering to mitigate hallucination in audio.

Adaptive vector steering: A training-free, layer-wise intervention for hallucination mitigation in large audio and multimodal models

TL;DR

The paper addresses hallucinations in large audio-language models by introducing Adaptive Vector Steering (AVS), a training-free, inference-time intervention that grounds model outputs in audio content. AVS rests on a layer-wise influence analysis showing later layers disproportionately shape final decisions, and it uses a contrastive steering vector to push activations toward audio grounding, with an adaptive scheme that strengthens later layers while preserving total steering strength. Empirical results on two diverse models (Qwen2-Audio-7B-Instruct and Gemma-3n-E4B-It) across Audio Hallucination QA and MMAU benchmarks demonstrate consistent improvements in F1 and accuracy, notably raising Gemma’s F1 to 0.619 and Qwen’s MMAU accuracy to 0.592 in the Test set. The work contributes a practical, interpretable, and generalizable mechanism for reducing ungrounded content in multimodalAudio-Language Models without fine-tuning, with potential implications for reliability and deployment of audio-grounded AI systems.

Abstract

Large Audio-Language Models and Multi-Modal Large Language Models have demonstrated strong capabilities in tasks such as Audio Question Answering (AQA), Audio Captioning, and Automatic Speech Recognition (ASR). However, there is growing evidence that these models can hallucinate about the content of the audio. To address this issue, we probe the models' internal states and propose Adaptive Vector Steering (AVS), a method that better grounds generation in audio content. We also identify a strong correlation between output correctness and internal representations. Experiments show consistent performance gains across two models and two benchmarks. On the Audio Hallucination QA dataset, our method boosts the F1-score of Gemma from 0.550 to 0.619 and Qwen from 0.626 to 0.632. Furthermore, our method increases the accuracy of Qwen on MMAU from 0.548 to 0.592, marking an 8% relative increase. To the best of our knowledge, this is the first work to apply vector steering to mitigate hallucination in audio.
Paper Structure (21 sections, 4 equations, 3 figures, 1 table)

This paper contains 21 sections, 4 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Steering Vectors for LLMs
  4. Hallucination in Audio Models
  5. Method
  6. Layer-wise Influence Analysis
  7. Vector Steering and Adaptive Vector Steering
  8. Vector Steering
  9. Adaptive Vector Steering
  10. Application During Inference
  11. Experment Setup
  12. Evaluated Models
  13. Model Choices
  14. Parameter Settings
  15. Benchmark
  16. ...and 6 more sections

Figures (3)

  • Figure 1: Illustration of Vector Steering The steering vector is defined as the contrast between the last-token hidden states of the residual streams, computed by subtracting the representation of the negative instance from that of the positive instance. Part of this figure is adapted from vis.
  • Figure 2: Analysis of Steering Vector Effects Across Qwen Model Layers. The left panel shows the cosine similarity for correct (green) and incorrect (red) vectors. The right panel displays the Cohen's d effect size for each layer. The later layers exhibit a more significant difference between correct and incorrect vectors, as seen in the cosine similarity difference.
  • Figure 3: Analysis of Steering Vector Effects Across Gemma Model Layers. Similar to Figure \ref{['fig:Qwen-layers']}. The later layers in Gemma also exhibit a more significant difference between correct and incorrect vectors, as seen in the cosine similarity difference.