Listen Like a Teacher: Mitigating Whisper Hallucinations using Adaptive Layer Attention and Knowledge Distillation

TL;DR

This work addresses hallucination-prone Whisper-based ASR under noisy conditions by proposing a two-stage framework: Stage-1 Adaptive Layer Attention (ALA) dynamically fuses encoder layer blocks to produce robust representations, and Stage-2 Multi-Objective Knowledge Distillation (MOKD) aligns a noisy-student decoder with a clean-teacher model through a multi-objective loss that includes encoder/decoder semantic alignment and cross-attention transfer. The approach yields significant improvements in both Word Error Rate and semantic fidelity (SeMaScore) across four languages and varying noise levels, while maintaining clean-speech performance. The results demonstrate that combining encoder-layer fusion with attention-aware distillation provides a principled route to mitigate hallucinations in real-world noisy ASR scenarios. This framework offers practical gains for reliable Whisper deployments and suggests avenues for cross-lingual generalization and application to other transformer-based ASR models.

Abstract

The Whisper model, an open-source automatic speech recognition system, is widely adopted for its strong performance across multilingual and zero-shot settings. However, it frequently suffers from hallucination errors, especially under noisy acoustic conditions. Previous works to reduce hallucinations in Whisper-style ASR systems have primarily focused on audio preprocessing or post-processing of transcriptions to filter out erroneous content. However, modifications to the Whisper model itself remain largely unexplored to mitigate hallucinations directly. To address this challenge, we present a two-stage architecture that first enhances encoder robustness through Adaptive Layer Attention (ALA) and further suppresses hallucinations using a multi-objective knowledge distillation (KD) framework. In the first stage, ALA groups encoder layers into semantically coherent blocks via inter-layer correlation analysis. A learnable multi-head attention module then fuses these block representations, enabling the model to jointly exploit low- and high-level features for more robust encoding. In the second stage, our KD framework trains the student model on noisy audio to align its semantic and attention distributions with a teacher model processing clean inputs. Our experiments on noisy speech benchmarks show notable reductions in hallucinations and word error rates, while preserving performance on clean speech. Together, ALA and KD offer a principled strategy to improve Whisper's reliability under real-world noisy conditions.

Figures (3)

• Figure 1: Block diagram of the proposed architecture combining Adaptive Layer Attention for encoder feature fusion and Multi-Objective Knowledge Distillation to reduce hallucination. $E_{final}$ and $D_{final}$ represents encoder and decoder final hidden states.
• Figure 2: Heatmap for encoder inter-layer similarity on a subset of Hindi Kathbath test dataset.
• Figure 3: Layer-wise attention behaviour for noisy speech input.