A Survey of Deep Learning for Complex Speech Spectrograms

TL;DR

This survey compiles the landscape of deep learning approaches for complex speech spectrograms, emphasizing both complex-valued and real-valued neural networks. It analyzes activation and normalization components, architectures across FFN/CNN/RNN/Transformer, and training strategies including low-latency and multi-domain losses. It synthesizes applications in phase retrieval, speech enhancement, and speaker separation, and surveys the growing role of generative models (VAEs, GANs, diffusion) in complex-spectrogram processing. The work highlights practical considerations, such as latency constraints and sub-band/time-frequency modeling, and positions CVNNs and RVNNs as complementary paths for exploiting complex-valued representations. Overall, it provides a comprehensive reference for researchers and practitioners pursuing high-quality, real-time speech processing in the complex-domain.

Abstract

Recent advancements in deep learning have significantly impacted the field of speech signal processing, particularly in the analysis and manipulation of complex spectrograms. This survey provides a comprehensive overview of the state-of-the-art techniques leveraging deep neural networks for processing complex spectrograms, which encapsulate both magnitude and phase information. We begin by introducing complex spectrograms and their associated features for various speech processing tasks. Next, we examine the key components and architectures of complex-valued neural networks, which are specifically designed to handle complex-valued data and have been applied to complex spectrogram processing. As recent studies have primarily focused on applying real-valued neural networks to complex spectrograms, we revisit these approaches and their architectural designs. We then discuss various training strategies and loss functions tailored for training neural networks to process and model complex spectrograms. The survey further examines key applications, including phase retrieval, speech enhancement, and speaker separation, where deep learning has achieved significant progress by leveraging complex spectrograms or their derived feature representations. Additionally, we examine the intersection of complex spectrograms with generative models. This survey aims to serve as a valuable resource for researchers and practitioners in the field of speech signal processing, deep learning and related fields.

Figures (4)

• Figure 1: This figure illustrates the overall structure of the survey, centered on the modeling and processing of complex spectrograms. The framework is divided into five main components: (1) complex-valued neural networks specifically designed for complex spectrogram processing; (2) RVNNs applied to complex spectrograms; (3) training strategies and loss functions, encompassing low-latency methods, optimization techniques, and loss designs tailored to complex spectrogram learning; (4) applications, covering speech enhancement, speaker separation, and phase retrieval; and (5) generative models, including approaches based on VAEs, GANs, and diffusion models for complex spectrogram processing.
• Figure 2: Features derived from one utterance in the TIMIT dataset (speaker ID: FVMH0, utterance ID: SA1). Time-frequency representations include log-magnitude, phase, real/imaginary components, group delay, and instantaneous frequency spectrograms. Computed using STFT with a 25 ms Hamming window and 10 ms frame shift. For clarity, real and imaginary spectrograms are displayed as absolute values on the logarithmic scale.
• Figure 3: Illustration of two common input representations for real-imaginary or magnitude-phase spectrograms in RVNN architectures. (a) Concatenated representation (Concatenated): The real and imaginary (or magnitude and phase) components of each complex-valued time–frequency vector are concatenated into a single vector, resulting in a higher-dimensional real-valued feature vector. (b) Stacked representation (Stacked): the real and imaginary (or magnitude and phase) components are stacked into a three-dimensional tensor with dimensions [Frequency, Time, 2], where the last dimension corresponds to the real and imaginary (or magnitude and phase) components.
• Figure 4: Illustration of two commonly used architectures for processing magnitude and phase spectrograms.