ReStyle-TTS: Relative and Continuous Style Control for Zero-Shot Speech Synthesis

TL;DR

ReStyle-TTS tackles the challenge of controllable zero-shot TTS by decoupling reference-style guidance from text guidance using Decoupled Classifier-Free Guidance, enabling continuous, relative control over pitch, energy, and emotions while preserving speaker timbre. It introduces style-specific LoRAs and Orthogonal LoRA Fusion to combine multiple attributes without interference, and Timbre Consistency Optimization to mitigate timbre drift when reducing reference guidance. The method demonstrates smooth single- and multi-attribute control, robust relative adjustments, and effective handling of contradictory-style scenarios where reference and target styles clash. This yields user-friendly, fine-grained control in zero-shot synthesis with practical applicability to mismatched references and expressive speech generation.

Abstract

Zero-shot text-to-speech models can clone a speaker's timbre from a short reference audio, but they also strongly inherit the speaking style present in the reference. As a result, synthesizing speech with a desired style often requires carefully selecting reference audio, which is impractical when only limited or mismatched references are available. While recent controllable TTS methods attempt to address this issue, they typically rely on absolute style targets and discrete textual prompts, and therefore do not support continuous and reference-relative style control. We propose ReStyle-TTS, a framework that enables continuous and reference-relative style control in zero-shot TTS. Our key insight is that effective style control requires first reducing the model's implicit dependence on reference style before introducing explicit control mechanisms. To this end, we introduce Decoupled Classifier-Free Guidance (DCFG), which independently controls text and reference guidance, reducing reliance on reference style while preserving text fidelity. On top of this, we apply style-specific LoRAs together with Orthogonal LoRA Fusion to enable continuous and disentangled multi-attribute control, and introduce a Timbre Consistency Optimization module to mitigate timbre drift caused by weakened reference guidance. Experiments show that ReStyle-TTS enables user-friendly, continuous, and relative control over pitch, energy, and multiple emotions while maintaining intelligibility and speaker timbre, and performs robustly in challenging mismatched reference-target style scenarios.

Figures (10)

• Figure 1: The overall framework of ReStyle-TTS. The proposed method consists of three logically coordinated components: (1) Decoupled Classifier-Free Guidance (DCFG) reduces the model's dependency on the reference style while maintaining text fidelity; (2) Orthogonal LoRA Fusion (OLoRA) reduces interference among multiple style-specific LoRAs by projecting each LoRA onto the orthogonal complement of the subspace spanned by the others; and (3) Timbre Consistency Optimization (TCO) reinforces speaker identity preservation through a similarity-based reward mechanism.
• Figure 2: Continuous single-attribute control with style-specific LoRAs.
• Figure 3: Two-attribute joint control results.
• Figure 4: Three-attribute joint control results.
• Figure 5: Reference energy vs. generated energy under different LoRA strengths.