CLARITY: Contextual Linguistic Adaptation and Accent Retrieval for Dual-Bias Mitigation in Text-to-Speech Generation

TL;DR

CLARITY tackles dual biases in instruction-guided TTS—accent bias from imbalanced training data and linguistic bias from user prompts—by introducing a backbone-agnostic two-signal framework: (i) contextual linguistic adaptation via an LLM to localize input text to the target dialect, and (ii) retrieval-augmented accent prompting (RAAP) to supply accent-consistent prompts. The objective combines $J_{\text{LLM}}(x^{*}, m)$, the LLM-based judgment of text adherence to metadata, with $C(s^{*}, m)$, the accent-consistency cue, under the synthesis constraint $\hat{y} = g_{\text{TTS}}(x^{*}, s^{*})$; structured metadata $m$ is inferred from user instructions through an LLM. CLARITY leverages text adaptation from multiple LLMs and RAAP from a curated prompt pool to achieve accent fidelity and dialect-aware content, demonstrated across twelve English accents with improvements in accent accuracy, fairness (lower bias), and perceptual quality (NISQA), while remaining backbone-agnostic and extensible to multilingual settings. This framework lays the groundwork for more inclusive TTS that respects both how language is spoken and how it is written, enabling more authentic, regionally appropriate synthetic speech in real-world deployments. $J_{\text{LLM}}(x^{*}, m) + C(s^{*}, m)$ governs the optimization, with $\hat{y} = g_{\text{TTS}}(x^{*}, s^{*})$ ensuring faithful synthesis.$

Abstract

Instruction-guided text-to-speech (TTS) research has reached a maturity level where excellent speech generation quality is possible on demand, yet two coupled biases persist: accent bias, where models default to dominant phonetic patterns, and linguistic bias, where dialect-specific lexical and cultural cues are ignored. These biases are interdependent, as authentic accent generation requires both accent fidelity and localized text. We present Contextual Linguistic Adaptation and Retrieval for Inclusive TTS sYnthesis (CLARITY), a backbone-agnostic framework that addresses these biases through dual-signal optimization: (i) contextual linguistic adaptation that localizes input text to the target dialect, and (ii) retrieval-augmented accent prompting (RAAP) that supplies accent-consistent speech prompts. Across twelve English accents, CLARITY improves accent accuracy and fairness while maintaining strong perceptual quality.

Figures (14)

• Figure 1: The "dual-bias" linguistic mismatch (prompt-source) with accent bias (system-source) reduces naturalness and authenticity.
• Figure 2: Our proposed CLARITY for mitigating user-side linguistic bias and system-side accent bias.
• Figure 3: RAAP accuracy (%) for gender, predicted accent, age attribute (left) and LLM-as-judge scores for various texts (right).
• Figure 4: Accent accuracy (%) (left) and NISQA (right) for the baselines and proposed CLARITY with GPT adapted text.
• Figure 5: Subjective Evaluation Results.