Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-band Frequency Reconstruction for Neural Psychoacoustic Coding

Dianwen Ng, Kun Zhou, Yi-Wen Chao, Zhiwei Xiong, Bin Ma, Eng Siong Chng

TL;DR

MUFFIN introduces a neural psychoacoustic codec that uses multi-band spectral RVQ to allocate bitrate according to perceptual salience, operating in a latent space with bands defined in the low-frequency region and a transformer-inspired convolutional backbone. The approach achieves unprecedented compression, including a state-of-the-art 12.5 Hz variant, while preserving perceptual quality and enabling zero-shot TTS with language models. Key contributions include the MBS-RVQ mechanism, a modified snake activation for improved spectral preservation, and extensive ablations showing codebook-specific roles in intelligibility and speaker disentanglement. The work demonstrates strong reconstruction performance across speech, music, and environmental sounds and highlights the potential for MUFFIN tokens to integrate with LLM-based systems, while addressing real-time efficiency and ethical considerations surrounding speaker privacy and synthesis safety.

Abstract

Achieving high-fidelity audio compression while preserving perceptual quality across diverse content remains a key challenge in Neural Audio Coding (NAC). We introduce MUFFIN, a fully convolutional Neural Psychoacoustic Coding (NPC) framework that leverages psychoacoustically guided multi-band frequency reconstruction. At its core is a Multi-Band Spectral Residual Vector Quantization (MBS-RVQ) module that allocates bitrate across frequency bands based on perceptual salience. This design enables efficient compression while disentangling speaker identity from content using distinct codebooks. MUFFIN incorporates a transformer-inspired convolutional backbone and a modified snake activation to enhance resolution in fine-grained spectral regions. Experimental results on multiple benchmarks demonstrate that MUFFIN consistently outperforms existing approaches in reconstruction quality. A high-compression variant achieves a state-of-the-art 12.5 Hz rate with minimal loss. MUFFIN also proves effective in downstream generative tasks, highlighting its promise as a token representation for integration with language models. Audio samples and code are available.

Multi-band Frequency Reconstruction for Neural Psychoacoustic Coding

TL;DR

MUFFIN introduces a neural psychoacoustic codec that uses multi-band spectral RVQ to allocate bitrate according to perceptual salience, operating in a latent space with bands defined in the low-frequency region and a transformer-inspired convolutional backbone. The approach achieves unprecedented compression, including a state-of-the-art 12.5 Hz variant, while preserving perceptual quality and enabling zero-shot TTS with language models. Key contributions include the MBS-RVQ mechanism, a modified snake activation for improved spectral preservation, and extensive ablations showing codebook-specific roles in intelligibility and speaker disentanglement. The work demonstrates strong reconstruction performance across speech, music, and environmental sounds and highlights the potential for MUFFIN tokens to integrate with LLM-based systems, while addressing real-time efficiency and ethical considerations surrounding speaker privacy and synthesis safety.

Abstract

Achieving high-fidelity audio compression while preserving perceptual quality across diverse content remains a key challenge in Neural Audio Coding (NAC). We introduce MUFFIN, a fully convolutional Neural Psychoacoustic Coding (NPC) framework that leverages psychoacoustically guided multi-band frequency reconstruction. At its core is a Multi-Band Spectral Residual Vector Quantization (MBS-RVQ) module that allocates bitrate across frequency bands based on perceptual salience. This design enables efficient compression while disentangling speaker identity from content using distinct codebooks. MUFFIN incorporates a transformer-inspired convolutional backbone and a modified snake activation to enhance resolution in fine-grained spectral regions. Experimental results on multiple benchmarks demonstrate that MUFFIN consistently outperforms existing approaches in reconstruction quality. A high-compression variant achieves a state-of-the-art 12.5 Hz rate with minimal loss. MUFFIN also proves effective in downstream generative tasks, highlighting its promise as a token representation for integration with language models. Audio samples and code are available.
Paper Structure (21 sections, 1 theorem, 8 equations, 7 figures, 8 tables)

This paper contains 21 sections, 1 theorem, 8 equations, 7 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methods: Multi-Band Frequency Coding
  4. Multi-band spectral residual vector quantization (MBS-RVQ)
  5. Analysis: Multi-band modeling improves generative quality by leveraging the perceptual entropy bound.
  6. Model Architecture
  7. Training Objectives
  8. Experiments
  9. Experimental Results
  10. Ablation Studies of Deconstructing MBS Codes
  11. Zero-shot TTS with MUFFIN tokens
  12. Conclusion
  13. Psychoacoustic evidence of perceptual speech characteristics
  14. Validity of MBS-RVQ at latent representation space.
  15. On the effect of MBS-RVQ compared to vanilla RVQ.
  16. ...and 6 more sections

Key Result

Theorem 3.1

cover1999elements For the given audio signal $\mathbf{x}(t)$ and compressed representation $\hat{\mathbf{x}}(t)$, the perceptual entropy $E_p$ satisfies the following lower bound when optimal multiband modeling is employed: where $H(B_k \,\vert\, \mathbf{x}(t))$ is the Shannon entropy of the signal components within band $B_k$, and $\Delta(B_k, \mathbf{x}(t))$ is the perceptual masking effect tha

Figures (7)

  • Figure 1: Illustration of the MBS-RVQ process: Fast Fourier Transform (FFT) is applied to the encoded latent representation to isolate specific frequency bands, capturing targeted spectral information for each codebook. The filtered representation is reconstructed using inverse FFT before undergoing quantization. The quantization residuals are then passed to the next codebook
  • Figure 2: Architecture of MUFFIN incorporating a fully convolutional structure.
  • Figure 3: The figures have been sourced from french1947factors, which discusses how speech sounds are recognized by the ear. The data were collected from microphones based on human speech and then analyzed with computational tools to derive the intensity and sound pressure levels. (1) Comparison of Speech Spectra. (2) Idealized Long Average Speech Spectrum at one meter from lips. (3) R.m.s. pressure of speech at 30cm from lips. (4) Articulation test with low pass filters. (5) Articulation test with high pass filters. (6) Syllable articulation versus cut-off frequency.
  • Figure 4: Illustration of our proposed modifications to the vanilla snake activation and its behavior in actual modeling for different sequential data.
  • Figure 5: A t-SNE plot showcasing each codebook, with speech randomly sampled from VoxCeleb, effectively represents six distinct speakers of the color code.
  • ...and 2 more figures

Theorems & Definitions (1)

  • Theorem 3.1: Perceptual Entropy and Masking Bounds