TASLA: Text-Aligned Speech Tokens with Multiple Layer-Aggregation

TL;DR

TASLA tackles the challenge of building text-aligned speech tokens that retain acoustic detail under aggressive compression. It introduces Multi-Layer Dynamic Attention (MLDA) to adaptively fuse shallow and deep encoder features per text position and Finite Scalar Quantization (FSQ) to discretize these features with minimal bitrate impact. Empirically, TASLA achieves a token rate of about $2.62$ Hz, delivers superior prosody metrics, and remains competitive on quality across LibriSpeech and out-of-domain datasets, while providing interpretable links between layer usage and spectral dynamics. The approach narrows the gap between alignment convenience and acoustic expressivity, enabling effective joint speech–text modeling for Spoken Language Models at low bitrates and short sequences.

Abstract

We propose Text-Aligned Speech Tokens with Multiple Layer-Aggregation (TASLA), which is a text-aligned speech tokenization framework that aims to address the problem that under a low-frame-rate and text-aligned regime, single-source speech tokens may lose acoustic details during reconstruction. On the other hand, this paper further explains how different encoder layers collaborate to capture comprehensive acoustic features for tokenization. Previous work, TASTE, proposed the text-aligned speech tokenization framework, which is a LM-friendly architecture, but struggles to capture acoustic details. We address this trade-off with two components: Multi-Layer Dynamic Attention (MLDA), which lets each text position adaptively mix shallow/deep features from a frozen speech encoder, and Finite Scalar Quantization (FSQ), a simple per-dimension discretization with smooth optimization. At about 2.62 Hz (tokens/s), TASLA consistently improves prosody and achieves competitive quality over TASTE on in-domain (LibriSpeech) and OOD (EXPRESSO, Voxceleb) sets. We further demonstrate that dynamic layer mixing is correlated with spectral flux and explains why MLDA preserves prosody under a low frame rate with extreme feature compression.

Figures (4)

• Figure 1: TASLA overview. Text tokens ($Q$) cross‑attend to the last‑layer encoder states ($K$) and to a dynamically mixed value stream ($V$) formed by per‑frame weights over layers 8/16/24/32 (MLDA). The aggregated features ($z$) are discretized by FSQ ($d=64$, $L=8$) and fed to a unit decoder that predicts S3 units; a vocoder reconstructs waveforms. Training minimizes next‑unit cross‑entropy ($\mathcal{L}_{CE}$) and an FSQ reconstruction loss $(\mathcal{L}_{rec})$.
• Figure 2: Scatter plots and linear regression across all samples. The figure shows that $w_1$ is positively correlated with mean spectral flux, while $w_2$ is negatively correlated. This indicates that $w_2$ suppresses spectral transients, whereas $w_1$ promotes them.
• Figure 3: S3 Unit Training Accuracy. Accuracy of S3 unit prediction for the three ablations: Text-only, TASTE, and TASLA. TASLA achieves about 10% higher accuracy than Text-only and TASTE.
• Figure 4: Layer-wise Weight Dynamics on Expresso. Four examples from Expresso showing Mel spectrograms, spectral flux, and dynamic layer weights ($w_0$–$w_3$). $w_3$ remains consistently small, while $w_0$, $w_1$, and $w_2$ dominate across most frames.