On the Fallacy of Global Token Perplexity in Spoken Language Model Evaluation

TL;DR

This paper argues that applying global token perplexity from text-modeling to spoken language models inadequately captures acoustic properties, potentially misrepresenting SLM progress. It introduces two evaluation families—localized/normalized likelihood-based metrics and generation-based evaluations with a model-as-a-judge—to better reflect speech characteristics and human perception. Empirical results on SALMon show stronger correlations with mean opinion scores and reveal a markedly different performance landscape, with top models closing a larger portion of the gap to human toplines (e.g., 83% for the leading model). The work offers a principled, scalable evaluation paradigm for spoken language modeling and highlights the importance of modality-aware assessment in future benchmarks.

Abstract

Generative spoken language models pretrained on large-scale raw audio can continue a speech prompt with appropriate content while preserving attributes like speaker and emotion, serving as foundation models for spoken dialogue. In prior literature, these models are often evaluated using ``global token perplexity'', which directly applies the text perplexity formulation to speech tokens. However, this practice overlooks fundamental differences between speech and text modalities, possibly leading to an underestimation of the speech characteristics. In this work, we propose a variety of likelihood- and generative-based evaluation methods that serve in place of naive global token perplexity. We demonstrate that the proposed evaluations more faithfully reflect perceived generation quality, as evidenced by stronger correlations with human-rated mean opinion scores (MOS). When assessed under the new metrics, the relative performance landscape of spoken language models is reshaped, revealing a significantly reduced gap between the best-performing model and the human topline. Together, these results suggest that appropriate evaluation is critical for accurately assessing progress in spoken language modeling.

Figures (13)

• Figure 1: Acoustic discontinuity disturbs negative log-likelihood loss (NLL) responses locally. Top: SALMon samples consist of a shared prompt and a separate continuation, where positive samples maintain acoustic consistency, and negative samples contain abrupt acoustic transitions. Bottom: NLL response of Llama-Mimi-1.3B with standard error margins. Response on negative samples show localized spike within a short temporal window after the transition in contrast to the positive sample. Global token perplexity aggregates likelihood contributions outside this localized region (Sec. \ref{['sec:method-global-ppl']}), making it susceptible to long-range loss volatility, thereby motivating our localized and normalized evaluation methods (Sec. \ref{['sec:method-altppl']}, \ref{['sec:method-cont']})
• Figure 2: NLL response of various models on SALMon samples with standard error margins. High-scoring models on SALMon (e.g., Flow-SLM) exhibit localized NLL spikes for negative samples within a short temporal window after the transition. This behavior is less apparent in lower-performing models such as GSLM.
• Figure 3: Overall performance of spoken language models on consistency tasks. The x-axis shows model accuracy (score) under different evaluators: (a1) alternative likelihood estimators, (b) MOS, and (c) embedding-as-a-judge, where model color codes are shown in (a1) and shared among all plots. In (a1), we correlate scores from proposed methods against those from global token perplexity (Global-PPL); the horizontal spread highlights the discrepancy across evaluation methods. The alternative methods rate strong models more favorably than Global-PPL, substantially closing the gap to the human topline. In (a2), we correlate deviations from the proposed methods against Global-PPL scores. Deviations generally become larger at higher Global-PPL performance (blue), until it saturates due to the maximum performance ceiling (orange). Negative deviations exhibit a similar trend in absolute magnitude, though this is less surprising since they are soft-bounded by distance to random baseline (green).
• Figure 4: Correlation between proposed evaluation methods vs golden labels provided by either MOS scores (top), or embedding judge proxies (bottom). The figure shows effectiveness of perplexity normalization and localization. The validility of the embedding judges stem from high correlation with the MOS scores (top right).
• Figure 5: Composition of the average per-sample advantage, which is defined as the difference between the negative loss and the positive loss. Advantage differs across evaluation methods in both token-type composition and loss magnitude.