On the Cost and Benefits of Training Context with Utterance or Full Conversation Training: A Comparative Stud

TL;DR

This paper tackles whether utterance-level training with contextual conditioning or full-conversation training yields better performance for conversational TTS under a fixed compute budget. Using the Dia framework and Honkai Star Rail dialogue data, the authors compare the two paradigms with a controlled 20 GPU-hour experiment, evaluating MOS, WER, speaker similarity, and context coherence. Results show utterance-level training achieves higher MOS (≈4.3) and lower WER, along with stronger speaker similarity, while full-conversation training incurs more memory and time costs and shows speaker-identity drift. The study provides practical guidelines favoring utterance-level training for resource-constrained, quality-sensitive conversational TTS and discusses hybrid approaches for future improvements.

Abstract

Modern TTS systems designed for conversations achieve high-quality utterances but often remain inaccessible publicly. Are existing open-source architectures inadequate, or are current training techniques insufficient? This paper investigates prominent models and their underlying behaviors regarding conversational context. Using 20 GPU-hours on an NVIDIA H100, we empirically examine two approaches: context-based utterance-level training versus full conversation training. Results demonstrate that context-based utterance training achieves superior MOS scores (4.3/5.0 vs 3.7/5.0) and reduces training time by 37%, while full conversation approaches suffer from speaker similarity hallucination issues. These findings provide practical guidelines for conversational TTS development, favoring utterance-level training with contextual conditioning for both resource efficiency and output quality.