Speech Slytherin: Examining the Performance and Efficiency of Mamba for Speech Separation, Recognition, and Synthesis

TL;DR

The paper evaluates Mamba, a linear-time state-space model, as an alternative to transformers across speech separation, ASR, and TTS. It introduces three models—Mamba-TasNet for separation, ConMamba for ASR, and VALL-M for TTS—and compares them to transformer baselines, including Sepformer, Conformer, and VALL-E. Across tasks, bidirectional Mamba encoders match or surpass transformer encoders in performance, while Mamba decoders can underperform transformer decoders in multimodal or autoregressive settings; importantly, Mamba shows memory and speed advantages for long speech sequences, with the degree of efficiency depending on token resolution. The study emphasizes that the superiority of Mamba versus transformers is task- and model-dependent, with clear use-case guidance for high-resolution separation versus multimodal reasoning, and it highlights the need for further architecture and hardware optimizations to fully realize Mamba’s potential in speech processing.

Abstract

It is too early to conclude that Mamba is a better alternative to transformers for speech before comparing Mamba with transformers in terms of both performance and efficiency in multiple speech-related tasks. To reach this conclusion, we propose and evaluate three models for three tasks: Mamba-TasNet for speech separation, ConMamba for speech recognition, and VALL-M for speech synthesis. We compare them with transformers of similar sizes in performance, memory, and speed. Our Mamba or Mamba-transformer hybrid models show comparable or higher performance than their transformer counterparts: Sepformer, Conformer, and VALL-E. They are more efficient than transformers in memory and speed for speech longer than a threshold duration, inversely related to the resolution of a speech token. Mamba for separation is the most efficient, and Mamba for recognition is the least. Further, we show that Mamba is not more efficient than transformer for speech shorter than the threshold duration and performs worse in models that require joint modeling of text and speech, such as cross or masked attention of two inputs. Therefore, we argue that the superiority of Mamba or transformer depends on particular problems and models. Code available at https://github.com/xi-j/Mamba-TasNet and https://github.com/xi-j/Mamba-ASR.

Figures (5)

• Figure 1: We propose three models for speech separation, recognition, and synthesis.
• Figure 2: The architecture of unidirectional and bidirectional Mamba.
• Figure 3: Mamba encoder, decoder, and ConMamba encoder layers. When used alone, the FeedForward module in the encoder and decoder is optional (none in Mamba-TasNet), and the CrossMamba module in the decoder is not needed.
• Figure 4: The architecture of Mamba-TasNet, ConMamba, and VALL-M.
• Figure 5: Comparisons of memory and speed for transformer and Mamba models in speech separation, recognition, and synthesis, with respect to speech inputs or outputs of different durations, benchmarked in an NVIDIA L40 GPU with 48GB memory.