Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Calm-Whisper: Reduce Whisper Hallucination On Non-Speech By Calming Crazy Heads Down

Yingzhi Wang, Anas Alhmoud, Saad Alsahly, Muhammad Alqurishi, Mirco Ravanelli

TL;DR

This work tackles Whisper's non-speech hallucination problem by first identifying which decoder self-attention heads drive hallucinations through head-wise masking, discovering that three heads account for the majority. It then performs calm-down fine-tuning by freezing all parameters except these three heads and training on non-speech data with blank labels, achieving a substantial reduction in non-speech hallucinations while preserving ASR accuracy. The approach uses non-speech datasets AudioSet, DEMAND, and MUSAN, and demonstrates an optimal balance around 5 fine-tuning epochs, culminating in Calm-Whisper with strong robustness to non-speech inputs and minimal WER degradation on LibriSpeech. The findings provide a practical path to improving industrial ASR reliability without relying on pre- or post-processing techniques.

Abstract

OpenAI's Whisper has achieved significant success in Automatic Speech Recognition. However, it has consistently been found to exhibit hallucination issues, particularly in non-speech segments, which limits its broader application in complex industrial settings. In this paper, we introduce a novel method to reduce Whisper's hallucination on non-speech segments without using any pre- or post-possessing techniques. Specifically, we benchmark the contribution of each self-attentional head in the Whisper-large-v3 decoder to the hallucination problem by performing a head-wise mask. Our findings reveal that only 3 of the 20 heads account for over 75% of the hallucinations on the UrbanSound dataset. We then fine-tune these three crazy heads using a collection of non-speech data. The results show that our best fine-tuned model, namely Calm-Whisper, achieves over 80% reduction in non-speech hallucination with only less than 0.1% WER degradation on LibriSpeech test-clean and test-other.

Calm-Whisper: Reduce Whisper Hallucination On Non-Speech By Calming Crazy Heads Down

TL;DR

This work tackles Whisper's non-speech hallucination problem by first identifying which decoder self-attention heads drive hallucinations through head-wise masking, discovering that three heads account for the majority. It then performs calm-down fine-tuning by freezing all parameters except these three heads and training on non-speech data with blank labels, achieving a substantial reduction in non-speech hallucinations while preserving ASR accuracy. The approach uses non-speech datasets AudioSet, DEMAND, and MUSAN, and demonstrates an optimal balance around 5 fine-tuning epochs, culminating in Calm-Whisper with strong robustness to non-speech inputs and minimal WER degradation on LibriSpeech. The findings provide a practical path to improving industrial ASR reliability without relying on pre- or post-processing techniques.

Abstract

OpenAI's Whisper has achieved significant success in Automatic Speech Recognition. However, it has consistently been found to exhibit hallucination issues, particularly in non-speech segments, which limits its broader application in complex industrial settings. In this paper, we introduce a novel method to reduce Whisper's hallucination on non-speech segments without using any pre- or post-possessing techniques. Specifically, we benchmark the contribution of each self-attentional head in the Whisper-large-v3 decoder to the hallucination problem by performing a head-wise mask. Our findings reveal that only 3 of the 20 heads account for over 75% of the hallucinations on the UrbanSound dataset. We then fine-tune these three crazy heads using a collection of non-speech data. The results show that our best fine-tuned model, namely Calm-Whisper, achieves over 80% reduction in non-speech hallucination with only less than 0.1% WER degradation on LibriSpeech test-clean and test-other.
Paper Structure (13 sections, 1 equation, 4 figures, 3 tables)

This paper contains 13 sections, 1 equation, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Whisper Hallucination On Non-Speech
  3. Evaluation Set
  4. Metric
  5. Results
  6. Hallucinatory Heads Detection
  7. Single-Head Masking
  8. Multi-Head Masking
  9. Calm-Down Fine-tuning
  10. Training set
  11. Experimental Settings
  12. Results and Discussion
  13. Conclusion

Figures (4)

  • Figure 1: Most frequently hallucinated sentences by Whisper-large-v3 on UrbanSound8K
  • Figure 2: Whisper-large-v3's hallucination rate on UrbanSound8K with one particular self-attention head in the decoder masked.
  • Figure 3: Calm-down fine-tuning designed to mitigate hallucinations in Whisper-large-v3. Specifically, all parameters in the Whisper-large-v3 model are frozen except for the parameters in head #1, #6, and #11 of each decoder layer. The model is trained using various non-speech noises paired with empty label strings to discourage it from reacting to non-speech inputs.
  • Figure 4: The evolution of the hallucination rate on UrbanSound8K and the WER on LibriSpeech test-clean across successive calm-down fine-tuning epochs for Whisper-large-v3.