Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Sylber 2.0: A Universal Syllable Embedding

Cheol Jun Cho, Nicholas Lee, Alan W Black, Gopala K. Anumanchipalli

TL;DR

Sylber 2.0 introduces a universal syllable-based self-supervised speech representation that compresses audio to roughly 5 Hz tokens while preserving linguistic and acoustic detail across languages and styles. The framework combines a content encoder learned via frame-wise and segmentation distillation, a boundary detector, a syllable-guided acoustic encoder, and a lightweight vocoder to achieve near-lossless reconstruction at 24 kHz, enabling efficient TTS with around 72M parameters and improved low-resource ASR. Key contributions include emergent multilingual syllabification, a compact yet expressive tokenization with separate content and acoustic information, and demonstrated downstream gains across TTS, SUPERB, and low-resource ASR. The results indicate Sylber 2.0 offers a scalable, language-agnostic syllabic abstraction that supports efficient multilingual speech modeling with strong reconstruction fidelity and practical downstream performance.

Abstract

Scaling spoken language modeling requires speech tokens that are both efficient and universal. Recent work has proposed syllables as promising speech tokens at low temporal resolution, but existing models are constrained to English and fail to capture sufficient acoustic detail. To address this gap, we present Sylber 2.0, a self-supervised framework for coding speech at the syllable level that enables efficient temporal compression and high-fidelity reconstruction. Sylber 2.0 achieves a very low token frequency around 5 Hz, while retaining both linguistic and acoustic detail across multiple languages and expressive styles. Experiments show that it performs on par with previous models operating on high-frequency baselines. Furthermore, Sylber 2.0 enables efficient TTS modeling which can generate speech with competitive intelligibility and quality with SOTA models using only 72M parameters. Moreover, the universality of Sylber 2.0 provides more effective features for low resource ASR than previous speech coding frameworks. In sum, we establish an effective syllable-level abstraction for general spoken language.

Sylber 2.0: A Universal Syllable Embedding

TL;DR

Sylber 2.0 introduces a universal syllable-based self-supervised speech representation that compresses audio to roughly 5 Hz tokens while preserving linguistic and acoustic detail across languages and styles. The framework combines a content encoder learned via frame-wise and segmentation distillation, a boundary detector, a syllable-guided acoustic encoder, and a lightweight vocoder to achieve near-lossless reconstruction at 24 kHz, enabling efficient TTS with around 72M parameters and improved low-resource ASR. Key contributions include emergent multilingual syllabification, a compact yet expressive tokenization with separate content and acoustic information, and demonstrated downstream gains across TTS, SUPERB, and low-resource ASR. The results indicate Sylber 2.0 offers a scalable, language-agnostic syllabic abstraction that supports efficient multilingual speech modeling with strong reconstruction fidelity and practical downstream performance.

Abstract

Scaling spoken language modeling requires speech tokens that are both efficient and universal. Recent work has proposed syllables as promising speech tokens at low temporal resolution, but existing models are constrained to English and fail to capture sufficient acoustic detail. To address this gap, we present Sylber 2.0, a self-supervised framework for coding speech at the syllable level that enables efficient temporal compression and high-fidelity reconstruction. Sylber 2.0 achieves a very low token frequency around 5 Hz, while retaining both linguistic and acoustic detail across multiple languages and expressive styles. Experiments show that it performs on par with previous models operating on high-frequency baselines. Furthermore, Sylber 2.0 enables efficient TTS modeling which can generate speech with competitive intelligibility and quality with SOTA models using only 72M parameters. Moreover, the universality of Sylber 2.0 provides more effective features for low resource ASR than previous speech coding frameworks. In sum, we establish an effective syllable-level abstraction for general spoken language.
Paper Structure (35 sections, 5 figures, 12 tables)

This paper contains 35 sections, 5 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methods
  4. Encoding-Decoding Framework with Syllabic Embeddings
  5. Self-Supervised Syllable Content Encoder
  6. Frame-Wise Self-Distillation
  7. Self-Segmentation Distillation
  8. Syllable-Guided Acoustic Encoder
  9. Vocoder for Syllable-to-Speech Synthesis
  10. Training Details
  11. Emergent Multilingual Syllabic Structure
  12. Near-Lossless Compression at 5 Hz
  13. Downstream Utilities
  14. Text-to-Speech through Syllable Embedding
  15. SUPERB Benchmark
  16. ...and 20 more sections

Figures (5)

  • Figure 1: Comparison of token frequency of speech and text tokenization methods. (left) Each bar indicates token frequency of Sylber 2.0 for each language from 102 languages in FLEURS-R. (right) Comparison with text BPE tokens. Each dot denotes each of 102 languages.
  • Figure 2: Encoding-decoding framework of Sylber 2.0. The model compresses speech into non-uniform ($\sim$ 5 Hz) embeddings with different components.
  • Figure 3: Similarity matrix after stage 1 and 3. The detected boundaries are denoted as red dashed lines.
  • Figure 4: Similarity matrix in three different languages using Sylber (top) and Sylber 2.0 (bottom). "*" denotes the segments which are masked out in the previous Sylber.
  • Figure 5: Visualization of Sylber 2.0 embeddings through tSNE. The top 50 syllables that are most corresponding to Sylber 2.0 are chosen and for each syllable 1K samples are retrieved from LibriSpeech. The top two rows use the content feature, with different colorization schemes for phonetic categories. The bottom uses the acoustic feature. While the manifold of the content features are phonetically structured, the acoustic feature shows no such structure.