Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SLICE: Speech Enhancement via Layer-wise Injection of Conditioning Embeddings

Seokhoon Moon, Kyudan Jung, Jaegul Choo

TL;DR

This work proposes injecting degradation conditioning, derived from a pretrained encoder with multi-task heads for noise type, reverberation, and distortion, into the timestep embedding so that it propagates through all residual blocks without architectural changes.

Abstract

Real-world speech is often corrupted by multiple degradations simultaneously, including additive noise, reverberation, and nonlinear distortion. Diffusion-based enhancement methods perform well on single degradations but struggle with compound corruptions. Prior noise-aware approaches inject conditioning at the input layer only, which can degrade performance below that of an unconditioned model. To address this, we propose injecting degradation conditioning, derived from a pretrained encoder with multi-task heads for noise type, reverberation, and distortion, into the timestep embedding so that it propagates through all residual blocks without architectural changes. In controlled experiments where only the injection method varies, input-level conditioning performs worse than no encoder at all on compound degradations, while layer-wise injection achieves the best results. The method also generalizes to diverse real-world recordings.

SLICE: Speech Enhancement via Layer-wise Injection of Conditioning Embeddings

TL;DR

This work proposes injecting degradation conditioning, derived from a pretrained encoder with multi-task heads for noise type, reverberation, and distortion, into the timestep embedding so that it propagates through all residual blocks without architectural changes.

Abstract

Real-world speech is often corrupted by multiple degradations simultaneously, including additive noise, reverberation, and nonlinear distortion. Diffusion-based enhancement methods perform well on single degradations but struggle with compound corruptions. Prior noise-aware approaches inject conditioning at the input layer only, which can degrade performance below that of an unconditioned model. To address this, we propose injecting degradation conditioning, derived from a pretrained encoder with multi-task heads for noise type, reverberation, and distortion, into the timestep embedding so that it propagates through all residual blocks without architectural changes. In controlled experiments where only the injection method varies, input-level conditioning performs worse than no encoder at all on compound degradations, while layer-wise injection achieves the best results. The method also generalizes to diverse real-world recordings.
Paper Structure (16 sections, 4 equations, 1 figure, 4 tables)

This paper contains 16 sections, 4 equations, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. Proposed Method
  3. Preliminaries: Score-Based Speech Enhancement
  4. Multi-Degradation Encoder
  5. Layer-wise Conditioning via Timestep Embedding Injection
  6. Training Objective
  7. Experiments
  8. Experimental Setup
  9. Datasets
  10. Baselines
  11. Metrics and Implementation
  12. Main Results
  13. Ablation Studies
  14. Analysis
  15. Conclusion
  16. ...and 1 more sections

Figures (1)

  • Figure 1: Overview of SLICE. (a) The encoder produces $\mathbf{h} \in \mathbb{R}^{d_h}$; branch projections are concatenated and mapped to $\mathbf{c}_\text{extra} \in \mathbb{R}^{d}$. (b)$\mathbf{c}_\text{extra}$ is added to $\mathbf{t}_\text{emb}$ and propagated through every residual block. Dashed orange: auxiliary heads (train-only). $d_w\!=\!768$, $d_h\!=\!256$, $d_b\!=\!128$, $d\!=\!512$.