Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Bone-conduction Guided Multimodal Speech Enhancement with Conditional Diffusion Models

Sina Khanagha, Bunlong Lay, Timo Gerkmann

TL;DR

This work tackles robust speech enhancement under severe noise by fusing noise-immune bone-conduction data with air-conducted audio through a conditional diffusion framework. It introduces the Bone-conduction Conditional Diffusion Model (BCDM) with two conditioning strategies (Input Concatenation and Decoder Conditioning) built on a modified NCSN++ backbone, and formulates the forward SDE $dx_t = f(x_t, t) dt + g(t) dW_t$ with a score model $s_ heta(x_t, y, y_c, t)$ trained via score matching. Experiments on the ABCS/CHiME3 setup show BCDM outperforms recent multimodal baselines and a strong single-modal diffusion baseline across a wide range of SNRs, with IC offering superior efficiency and DC delivering higher instrumental metrics at higher cost. This demonstrates the viability of conditional diffusion for bone-conduction guided speech enhancement and suggests practical gains for robust hearing aids and communication systems. Overall, the work establishes a new multimodal, diffusion-based paradigm that exploits complementary bone-conduction signals to overcome noise in single-channel scenarios.

Abstract

Single-channel speech enhancement models face significant performance degradation in extremely noisy environments. While prior work has shown that complementary bone-conducted speech can guide enhancement, effective integration of this noise-immune modality remains a challenge. This paper introduces a novel multimodal speech enhancement framework that integrates bone-conduction sensors with air-conducted microphones using a conditional diffusion model. Our proposed model significantly outperforms previously established multimodal techniques and a powerful diffusion-based single-modal baseline across a wide range of acoustic conditions.

Bone-conduction Guided Multimodal Speech Enhancement with Conditional Diffusion Models

TL;DR

This work tackles robust speech enhancement under severe noise by fusing noise-immune bone-conduction data with air-conducted audio through a conditional diffusion framework. It introduces the Bone-conduction Conditional Diffusion Model (BCDM) with two conditioning strategies (Input Concatenation and Decoder Conditioning) built on a modified NCSN++ backbone, and formulates the forward SDE with a score model trained via score matching. Experiments on the ABCS/CHiME3 setup show BCDM outperforms recent multimodal baselines and a strong single-modal diffusion baseline across a wide range of SNRs, with IC offering superior efficiency and DC delivering higher instrumental metrics at higher cost. This demonstrates the viability of conditional diffusion for bone-conduction guided speech enhancement and suggests practical gains for robust hearing aids and communication systems. Overall, the work establishes a new multimodal, diffusion-based paradigm that exploits complementary bone-conduction signals to overcome noise in single-channel scenarios.

Abstract

Single-channel speech enhancement models face significant performance degradation in extremely noisy environments. While prior work has shown that complementary bone-conducted speech can guide enhancement, effective integration of this noise-immune modality remains a challenge. This paper introduces a novel multimodal speech enhancement framework that integrates bone-conduction sensors with air-conducted microphones using a conditional diffusion model. Our proposed model significantly outperforms previously established multimodal techniques and a powerful diffusion-based single-modal baseline across a wide range of acoustic conditions.
Paper Structure (10 sections, 6 equations, 2 figures, 1 table)

This paper contains 10 sections, 6 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Method
  3. Conditioning Strategies
  4. Input Concatenation (IC)
  5. Decoder Conditioning (DC)
  6. Experimental Setup
  7. Results and Discussions
  8. Comparison Against Baseline Models
  9. Number of Reverse Steps
  10. Conclusion

Figures (2)

  • Figure 1: An overview of two IC (red) and DC (blue) conditioning strategies for the large variant of BCDM. The skip connections and progressive layers of the NCSN++ have been excluded from this diagram for clarity improvement
  • Figure 2: PESQ evolution of BCDM's small variants as a function of $N$ at -5 dB SNR