Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SPAM: Style Prompt Adherence Metric for Prompt-based TTS

Chanhee Cho, Nayeon Kim, Bugeun Kim

TL;DR

SPAM tackles the lack of reliable automatic metrics for prompt-based TTS adherence by introducing a CLAP-based metric with explicit acoustic-attribute factorization and a SupCon training objective. The architecture includes a WavLM-based speech encoder, an X-Vector speaker module, a G2P transcript encoder, and a Llama-3.1 prompt encoder, all feeding a four-branch speech-prompt fusion that yields a cosine similarity score with the prompt embedding. Plausibility and faithfulness experiments show SPAM correlates with human MOS and reliably distinguishes between semantically similar and dissimilar prompts, outperforming RA-CLAP baselines and generalizing to unseen prompts. The work offers a practical automatic evaluation tool for prompt adherence in TTS, enabling more robust model comparison and development.

Abstract

Prompt-based text-to-speech (TTS) aims to generate speech that adheres to fine-grained style cues provided in a text prompt. However, most prior works depend on neither plausible nor faithful measures to evaluate prompt adherence. That is, they cannot ensure whether the evaluation is grounded on the prompt and is similar to a human. Thus, we present a new automatic metric, the Style Prompt Adherence Metric, which explicitly satisfies both plausibility and faithfulness. Inspired by the CLAP, our approach factorizes speech into acoustic attributes and aligns them with the style prompt. Also, we trained the scorer with a supervised contrastive loss, which could provide a clearer distinction between different semantics. We conducted two experiments on two perspectives. The plausibility experiment showed that SPAM achieved a strong correlation with the mean opinion score (MOS). Also, the faithfulness experiment demonstrated that SPAM is successfully grounded to the given style prompt, as it can discriminate different semantics of the prompt. We believe that SPAM can provide a viable automatic solution for evaluating style prompt adherence of synthesized speech.

SPAM: Style Prompt Adherence Metric for Prompt-based TTS

TL;DR

SPAM tackles the lack of reliable automatic metrics for prompt-based TTS adherence by introducing a CLAP-based metric with explicit acoustic-attribute factorization and a SupCon training objective. The architecture includes a WavLM-based speech encoder, an X-Vector speaker module, a G2P transcript encoder, and a Llama-3.1 prompt encoder, all feeding a four-branch speech-prompt fusion that yields a cosine similarity score with the prompt embedding. Plausibility and faithfulness experiments show SPAM correlates with human MOS and reliably distinguishes between semantically similar and dissimilar prompts, outperforming RA-CLAP baselines and generalizing to unseen prompts. The work offers a practical automatic evaluation tool for prompt adherence in TTS, enabling more robust model comparison and development.

Abstract

Prompt-based text-to-speech (TTS) aims to generate speech that adheres to fine-grained style cues provided in a text prompt. However, most prior works depend on neither plausible nor faithful measures to evaluate prompt adherence. That is, they cannot ensure whether the evaluation is grounded on the prompt and is similar to a human. Thus, we present a new automatic metric, the Style Prompt Adherence Metric, which explicitly satisfies both plausibility and faithfulness. Inspired by the CLAP, our approach factorizes speech into acoustic attributes and aligns them with the style prompt. Also, we trained the scorer with a supervised contrastive loss, which could provide a clearer distinction between different semantics. We conducted two experiments on two perspectives. The plausibility experiment showed that SPAM achieved a strong correlation with the mean opinion score (MOS). Also, the faithfulness experiment demonstrated that SPAM is successfully grounded to the given style prompt, as it can discriminate different semantics of the prompt. We believe that SPAM can provide a viable automatic solution for evaluating style prompt adherence of synthesized speech.
Paper Structure (12 sections, 2 equations, 2 figures, 2 tables)

This paper contains 12 sections, 2 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. The SPAM Metric
  3. Speech Encoder
  4. Prompt Encoder
  5. Speech-Prompt Fusion Module
  6. Training Loss
  7. EXPERIMENTS
  8. Plausibility experiment
  9. Faithfulness experiment
  10. Datasets and Baselines
  11. RESULT AND DISCUSSION
  12. Conclusion

Figures (2)

  • Figure 1: Architecture of SPAM
  • Figure 2: Plausibility and Faithfulness Experiment