Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Speaking Clearly: A Simplified Whisper-Based Codec for Low-Bitrate Speech Coding

Xin Zhang, Lin Li, Xiangni Lu, Jianquan Liu, Kong Aik Lee

TL;DR

This work tackles the semantic-acoustic conflict in low-bitrate speech codecs by flipping the conventional approach: instead of adding semantic supervision to acoustic codecs, it adapts a semantically rich Whisper encoder for high-fidelity reconstruction through targeted architectural simplifications. The authors identify that removing the convolutional front-end GELU nonlinearity and absolute positional encodings unlocks Whisper's acoustic modeling while preserving its semantic grounding, culminating in SimWhisper-Codec. Using a frozen simplified encoder, FSQ quantization, and a symmetric decoder trained with a single-stage GAN objective, the codec achieves 1.1 kbps with competitive acoustic quality (PESQ-NB 2.98, STOI 0.91, SIM 0.83) and superior semantic preservation (WER 3.10) without external supervision, outperforming semantically-supervised baselines. The results suggest a practical semantic-first design paradigm for speech codecs with implications for low-bandwidth speech communication and speech-LMM integration.

Abstract

Speech codecs serve as bridges between continuous speech signals and large language models, yet face an inherent conflict between acoustic fidelity and semantic preservation. To mitigate this conflict, prevailing methods augment acoustic codecs with complex semantic supervision. We explore the opposite direction: a semantic-first approach that starts from a semantically-capable model and adapts it for high-fidelity acoustic reconstruction. Through empirical analysis, we discover that targeted architectural simplification can unlock the acoustic modeling potential of Whisper, a text-aligned Automatic Speech Recognition (ASR) model. Based on this finding, we propose SimWhisper-Codec, a novel codec that balances the semantic and acoustic preservation by leveraging a frozen, simplified Whisper encoder without requiring external supervision. Experimental results demonstrate that SimWhisper-Codec achieves superior performance in both semantic preservation and acoustic quality compared to semantically-supervised codecs such as Mimi Codec and SpeechTokenizer at similar bitrates, validating the effectiveness of our semantic-first approach. Code is available at https://github.com/ZhangXinWhut/SimWhisper-Codec.

Speaking Clearly: A Simplified Whisper-Based Codec for Low-Bitrate Speech Coding

TL;DR

This work tackles the semantic-acoustic conflict in low-bitrate speech codecs by flipping the conventional approach: instead of adding semantic supervision to acoustic codecs, it adapts a semantically rich Whisper encoder for high-fidelity reconstruction through targeted architectural simplifications. The authors identify that removing the convolutional front-end GELU nonlinearity and absolute positional encodings unlocks Whisper's acoustic modeling while preserving its semantic grounding, culminating in SimWhisper-Codec. Using a frozen simplified encoder, FSQ quantization, and a symmetric decoder trained with a single-stage GAN objective, the codec achieves 1.1 kbps with competitive acoustic quality (PESQ-NB 2.98, STOI 0.91, SIM 0.83) and superior semantic preservation (WER 3.10) without external supervision, outperforming semantically-supervised baselines. The results suggest a practical semantic-first design paradigm for speech codecs with implications for low-bandwidth speech communication and speech-LMM integration.

Abstract

Speech codecs serve as bridges between continuous speech signals and large language models, yet face an inherent conflict between acoustic fidelity and semantic preservation. To mitigate this conflict, prevailing methods augment acoustic codecs with complex semantic supervision. We explore the opposite direction: a semantic-first approach that starts from a semantically-capable model and adapts it for high-fidelity acoustic reconstruction. Through empirical analysis, we discover that targeted architectural simplification can unlock the acoustic modeling potential of Whisper, a text-aligned Automatic Speech Recognition (ASR) model. Based on this finding, we propose SimWhisper-Codec, a novel codec that balances the semantic and acoustic preservation by leveraging a frozen, simplified Whisper encoder without requiring external supervision. Experimental results demonstrate that SimWhisper-Codec achieves superior performance in both semantic preservation and acoustic quality compared to semantically-supervised codecs such as Mimi Codec and SpeechTokenizer at similar bitrates, validating the effectiveness of our semantic-first approach. Code is available at https://github.com/ZhangXinWhut/SimWhisper-Codec.
Paper Structure (15 sections, 5 equations, 3 figures, 3 tables)

This paper contains 15 sections, 5 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. EMPIRICAL ANALYSIS OF COMPONENTS HINDERING ACOUSTIC RECONSTRUCTION
  3. Component Analysis and Hypotheses
  4. Validation Analysis
  5. Method
  6. SimWhisper-Codec
  7. Model Architecture
  8. Training Objective
  9. Experiments
  10. Settings
  11. Evaluation Metrics
  12. Experimental Results
  13. Ablation Study
  14. Preservation of Acoustic Attributes
  15. Conclusion

Figures (3)

  • Figure 1: Overview of SimWhisper-Codec. A simplified Whisper encoder with FSQ discretization and a symmetric decoder. Model architecture and training procedure are detailed in Section \ref{['ssec:method']}.
  • Figure 2: Self-attention maps (log-weights, head-averaged) from a middle Transformer layer for the utterance "one, two, three, four, four, three, two, one". The maps demonstrate how absolute positional encodings affect attention patterns in repetitive structures.
  • Figure 3: Pitch tracking performance.