Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

mmWave Radar Aware Dual-Conditioned GAN for Speech Reconstruction of Signals With Low SNR

Jash Karani, Adithya Chittem, Deepan Roy, Sandeep Joshi

TL;DR

This work proposes a two-stage speech reconstruction pipeline for mmWave using a Radar-Aware Dual-conditioned Generative Adversarial Network (RAD-GAN), which is capable of performing bandwidth extension on signals with low signal-to-noise ratios, captured through glass walls.

Abstract

Millimeter-wave (mmWave) radar captures are band-limited and noisy, making for difficult reconstruction of intelligible full-bandwidth speech. In this work, we propose a two-stage speech reconstruction pipeline for mmWave using a Radar-Aware Dual-conditioned Generative Adversarial Network (RAD-GAN), which is capable of performing bandwidth extension on signals with low signal-to-noise ratios (-5 dB to -1 dB), captured through glass walls. We propose an mmWave-tailored Multi-Mel Discriminator (MMD) and a Residual Fusion Gate (RFG) to enhance the generator input to process multiple conditioning channels. The proposed two-stage pipeline involves pretraining the model on synthetically clipped clean speech and finetuning on fused mel spectrograms generated by the RFG. We empirically show that the proposed method, trained on a limited dataset, with no pre-trained modules, and no data augmentations, outperformed state-of-the-art approaches for this specific task. Audio examples of RAD-GAN are available online at https://rad-gan-demo-site.vercel.app/.

mmWave Radar Aware Dual-Conditioned GAN for Speech Reconstruction of Signals With Low SNR

TL;DR

This work proposes a two-stage speech reconstruction pipeline for mmWave using a Radar-Aware Dual-conditioned Generative Adversarial Network (RAD-GAN), which is capable of performing bandwidth extension on signals with low signal-to-noise ratios, captured through glass walls.

Abstract

Millimeter-wave (mmWave) radar captures are band-limited and noisy, making for difficult reconstruction of intelligible full-bandwidth speech. In this work, we propose a two-stage speech reconstruction pipeline for mmWave using a Radar-Aware Dual-conditioned Generative Adversarial Network (RAD-GAN), which is capable of performing bandwidth extension on signals with low signal-to-noise ratios (-5 dB to -1 dB), captured through glass walls. We propose an mmWave-tailored Multi-Mel Discriminator (MMD) and a Residual Fusion Gate (RFG) to enhance the generator input to process multiple conditioning channels. The proposed two-stage pipeline involves pretraining the model on synthetically clipped clean speech and finetuning on fused mel spectrograms generated by the RFG. We empirically show that the proposed method, trained on a limited dataset, with no pre-trained modules, and no data augmentations, outperformed state-of-the-art approaches for this specific task. Audio examples of RAD-GAN are available online at https://rad-gan-demo-site.vercel.app/.
Paper Structure (7 sections, 8 equations, 2 figures, 2 tables)

This paper contains 7 sections, 8 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. System Model
  3. RAD-GAN Architecture
  4. Experimental Setup
  5. Results and Analysis
  6. Conclusion
  7. Generative AI Use Disclosure

Figures (2)

  • Figure 1: Block diagram of the proposed mmWave radar to speech reconstruction pipeline.
  • Figure 2: Qualitative comparison on Task 2. Columns from left to right are: clean reference, noisy input, WVN output, and RAD-GAN output. The top row shows the time-domain waveforms, and the bottom row shows the corresponding spectrograms.