Dimension-First Evaluation of Speech-to-Speech Models with Structured Acoustic Cues

TL;DR

The paper tackles the challenge of evaluating speech-to-speech systems by moving beyond transcript-only LLM judgments and expensive Audio Language Models, introducing TRACE, a two-stage, training-free evaluator that reason over structured audio cues. TRACE uses Human Chain-of-Thought to collect dimension-wise judgments across Content, Voice Quality, and Paralinguistics, then fuses these through a deterministic policy to produce an overall score that aligns with human judgments while remaining cost-efficient.Empirical results on SpeakBench and S2S-Arena show TRACE achieves higher agreement with human annotations than baselines, with notable improvements in paralinguistics and robust performance across backbones, while significantly reducing cost.The framework is designed to be auditable and scalable, with plans to extend to multilingual settings, richer prosodic features, and real-time evaluation, while acknowledging limitations and ethical considerations in automated evaluation.

Abstract

Large Language Model (LLM) judges exhibit strong reasoning capabilities but are limited to textual content. This leaves current automatic Speech-to-Speech (S2S) evaluation methods reliant on opaque and expensive Audio Language Models (ALMs). In this work, we propose TRACE (Textual Reasoning over Audio Cues for Evaluation), a novel framework that enables LLM judges to reason over audio cues to achieve cost-efficient and human-aligned S2S evaluation. To demonstrate the strength of the framework, we first introduce a Human Chain-of-Thought (HCoT) annotation protocol to improve the diagnostic capability of existing judge benchmarks by separating evaluation into explicit dimensions: content (C), voice quality (VQ), and paralinguistics (P). Using this data, TRACE constructs a textual blueprint of inexpensive audio signals and prompts an LLM to render dimension-wise judgments, fusing them into an overall rating via a deterministic policy. TRACE achieves higher agreement with human raters than ALMs and transcript-only LLM judges while being significantly more cost-effective. We will release the HCoT annotations and the TRACE framework to enable scalable and human-aligned S2S evaluation.

Figures (9)

• Figure 1: Bridging the Evaluation Gap.(A) The Status Quo: Current transcript-only LLM judges are blind to paralinguistics whereas ALM judges are opaque and expensive, and benchmarks often force winners on noisy data. (B) TRACE Framework: We introduce HCoT, a dimension-first speech labeling protocol, and TRACE. TRACE extracts acoustic features into a structured blueprint, allowing text-LLMs to reason over audio for S2S evaluation.
• Figure 2: The TRACE Architecture.Phase 1 (Signal Extraction): We extract inexpensive signals for Content (ASR), Voice Quality (MOS predictors), and Paralinguistics (prosody, affect, energy etc.). Phase 2 (Inference): These signals form a structured textual blueprint of audio cues, which is then passed to an LLM judge to make dimension-wise decisions. The dimension-wise deicisons are fused via a deterministic tree to yield the final score.
• Figure 3: P1 Counterfactual (SpeakBench). TRACE selectively uses delivery (VQ) to break semantic ties (VQ share $\approx$23% vs. $\sim$3-5% for baselines).
• Figure 4: P2 Flip Rates (S2S-Arena). TRACE is significantly more sensitive to Paralinguistics than LLM-Judge or Audio Judge.
• Figure 5: P3 Attributing Performance (S2S-Arena). With many both-bad pairs (58%), TRACE cuts winner-on-bad from $\sim$70–74% to 48.6%.