Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient Speech Language Modeling via Energy Distance in Continuous Latent Space

Zhengrui Ma, Yang Feng, Chenze Shao, Fandong Meng, Jie Zhou, Min Zhang

TL;DR

SLED proposes a continuous latent-space autoregressive framework for speech language modeling that replaces discrete tokenization with an energy-distance objective to capture per-step distributions. By encoding waveforms with Encodec and using a lightweight per-step generator conditioned on autoregressive state, SLED achieves strong zero-shot and streaming speech synthesis without RVQ hierarchies. The approach yields competitive or superior results to discrete baselines, with demonstrated efficiency advantages and robust CFG-based generation. This work advances practical continuous autoregressive speech modeling and highlights pathways for general-purpose speech language models while underscoring considerations for voice cloning and safety.

Abstract

We introduce SLED, an alternative approach to speech language modeling by encoding speech waveforms into sequences of continuous latent representations and modeling them autoregressively using an energy distance objective. The energy distance offers an analytical measure of the distributional gap by contrasting simulated and target samples, enabling efficient training to capture the underlying continuous autoregressive distribution. By bypassing reliance on residual vector quantization, SLED avoids discretization errors and eliminates the need for the complicated hierarchical architectures common in existing speech language models. It simplifies the overall modeling pipeline while preserving the richness of speech information and maintaining inference efficiency. Empirical results demonstrate that SLED achieves strong performance in both zero-shot and streaming speech synthesis, showing its potential for broader applications in general-purpose speech language models.

Efficient Speech Language Modeling via Energy Distance in Continuous Latent Space

TL;DR

SLED proposes a continuous latent-space autoregressive framework for speech language modeling that replaces discrete tokenization with an energy-distance objective to capture per-step distributions. By encoding waveforms with Encodec and using a lightweight per-step generator conditioned on autoregressive state, SLED achieves strong zero-shot and streaming speech synthesis without RVQ hierarchies. The approach yields competitive or superior results to discrete baselines, with demonstrated efficiency advantages and robust CFG-based generation. This work advances practical continuous autoregressive speech modeling and highlights pathways for general-purpose speech language models while underscoring considerations for voice cloning and safety.

Abstract

We introduce SLED, an alternative approach to speech language modeling by encoding speech waveforms into sequences of continuous latent representations and modeling them autoregressively using an energy distance objective. The energy distance offers an analytical measure of the distributional gap by contrasting simulated and target samples, enabling efficient training to capture the underlying continuous autoregressive distribution. By bypassing reliance on residual vector quantization, SLED avoids discretization errors and eliminates the need for the complicated hierarchical architectures common in existing speech language models. It simplifies the overall modeling pipeline while preserving the richness of speech information and maintaining inference efficiency. Empirical results demonstrate that SLED achieves strong performance in both zero-shot and streaming speech synthesis, showing its potential for broader applications in general-purpose speech language models.
Paper Structure (22 sections, 12 equations, 3 figures, 7 tables)

This paper contains 22 sections, 12 equations, 3 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Maximum Mean Discrepancy and Generalized Energy Distance
  3. Approach
  4. Language Modeling in Continuous Latent Space
  5. Per-token Generative Modeling via Energy Distance
  6. Classifier-free Guidance
  7. Streaming Inference
  8. Related Work
  9. Experiments
  10. Training Datasets
  11. Experimental Settings
  12. Evaluation Settings
  13. Main Results
  14. Analysis
  15. Importance of Energy Distance
  16. ...and 7 more sections

Figures (3)

  • Figure 1: Different speech language modeling approaches. Left: VALL-E-style hierarchical architecture for RVQ token sequences. Middle: RQ-Transformer-style hierarchical architecture for RVQ token sequences. Right: Architecture for continuous token sequences.
  • Figure 2: Illustration of our streaming inference mechanism. Text and speech tokens are interleaved based on a predefined ratio, and the loss is computed only at positions where targets are shown.
  • Figure 3: Evaluation of classifier-free guidance effects across generation settings.