Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Whisper-CD: Accurate Long-Form Speech Recognition using Multi-Negative Contrastive Decoding

Hoseong Ahn, Jeongyun Chae, Yoonji Park, Kyuhong Shim

TL;DR

Whisper-CD is proposed, a training-free contrastive decoding framework that contrasts clean-audio logits against negative logits computed from three acoustically motivated perturbations: Gaussian noise injection, silence signal, and audio temporal shift, building a unified multi-negative objective for token-by-token decoding.

Abstract

Long-form speech recognition with large encoder-decoder models such as Whisper often exhibit hallucinations, repetition loops, and content omissions. These errors can accumulate and be further amplified when the previous segment's transcription is used as decoding context. We propose Whisper-CD, a training-free contrastive decoding framework that contrasts clean-audio logits against negative logits computed from three acoustically motivated perturbations: Gaussian noise injection, silence signal, and audio temporal shift. We aggregate these negatives via the log-sum-exp operator, building a unified multi-negative objective for token-by-token decoding. Across five English long-form benchmarks, Whisper-CD reduces WER by up to 24.3pp on CORAAL and shows 48% faster token generation throughput than beam search. Because Whisper-CD operates purely at inference time, it can be applied as a drop-in replacement to already-deployed Whisper systems without retraining.

Whisper-CD: Accurate Long-Form Speech Recognition using Multi-Negative Contrastive Decoding

TL;DR

Whisper-CD is proposed, a training-free contrastive decoding framework that contrasts clean-audio logits against negative logits computed from three acoustically motivated perturbations: Gaussian noise injection, silence signal, and audio temporal shift, building a unified multi-negative objective for token-by-token decoding.

Abstract

Long-form speech recognition with large encoder-decoder models such as Whisper often exhibit hallucinations, repetition loops, and content omissions. These errors can accumulate and be further amplified when the previous segment's transcription is used as decoding context. We propose Whisper-CD, a training-free contrastive decoding framework that contrasts clean-audio logits against negative logits computed from three acoustically motivated perturbations: Gaussian noise injection, silence signal, and audio temporal shift. We aggregate these negatives via the log-sum-exp operator, building a unified multi-negative objective for token-by-token decoding. Across five English long-form benchmarks, Whisper-CD reduces WER by up to 24.3pp on CORAAL and shows 48% faster token generation throughput than beam search. Because Whisper-CD operates purely at inference time, it can be applied as a drop-in replacement to already-deployed Whisper systems without retraining.
Paper Structure (20 sections, 3 equations, 2 figures, 4 tables)

This paper contains 20 sections, 3 equations, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Challenges in Long-form ASR
  4. Hallucinations in Whisper ASR
  5. Test-time Improvement in ASR
  6. Whisper-CD
  7. Motivation
  8. Contrastive Decoding for ASR
  9. Perturbation Strategies
  10. Gaussian Noise Injection.
  11. Silence Signal.
  12. Audio Temporal Shift.
  13. Multi-Negative Contrastive Decoding
  14. Inference
  15. Experimental Results
  16. ...and 5 more sections

Figures (2)

  • Figure 1: Overview of the proposed Whisper-CD. Each audio segment is processed through four parallel paths, comprising the original signal and three acoustically perturbed variants (Gaussian noise injection, silence signal, and audio temporal shift). Each path produces decoder logits conditioned on the corresponding encoder output, and contrastive decoding steers token selection away from hallucinated outputs such as repetition loops and content omissions.
  • Figure 2: Qualitative examples on the same audio inputs. The baseline falls into repetition loops (red), while Whisper-CD breaks the loop and recovers the correct transcription (green).