Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CapTalk: Unified Voice Design for Single-Utterance and Dialogue Speech Generation

Xiaosu Su, Zihan Sun, Peilei Jia, Jun Gao

Abstract

Voice design from natural language descriptions is emerging as a new task in text-to-speech multimodal generation, aiming to synthesize speech with target timbre and speaking style without relying on reference audio. However, existing methods mainly focus on single-utterance generation, leaving conversational voice design largely unexplored. In this work, we extend voice design to dialogue, enabling better target speaker modeling and turn-level expressive control in natural conversational settings. We propose CapTalk, a unified caption-conditioned text-audio autoregressive framework for both single-utterance and dialogue voice design. CapTalk uses utterance-level captions for single-utterance voice design and speaker-level captions for dialogue speaker modeling, and further introduces a CoT control sequence in dialogue to explicitly plan turn-level dynamic attributes. To resolve the conflict between stable timbre preservation and context-adaptive expression, we propose a hierarchical variational conditioning module with an utterance-level speaker encoder to better balance stable timbre preservation and context-adaptive expression. This enables timbre reuse while keeping expression adaptive to the current utterance and, in dialogue, the surrounding context. We also build a comprehensive evaluation protocol for both single-utterance and dialogue settings. Experiments show that CapTalk achieves state-of-the-art performance on a single-utterance voice design benchmark and delivers better expression controllability and contextual appropriateness in multi-turn dialogue. Audio samples are available at: https://anonymous.4open.science/api/repo/Captalk-D601/file/index.html.

CapTalk: Unified Voice Design for Single-Utterance and Dialogue Speech Generation

Abstract

Voice design from natural language descriptions is emerging as a new task in text-to-speech multimodal generation, aiming to synthesize speech with target timbre and speaking style without relying on reference audio. However, existing methods mainly focus on single-utterance generation, leaving conversational voice design largely unexplored. In this work, we extend voice design to dialogue, enabling better target speaker modeling and turn-level expressive control in natural conversational settings. We propose CapTalk, a unified caption-conditioned text-audio autoregressive framework for both single-utterance and dialogue voice design. CapTalk uses utterance-level captions for single-utterance voice design and speaker-level captions for dialogue speaker modeling, and further introduces a CoT control sequence in dialogue to explicitly plan turn-level dynamic attributes. To resolve the conflict between stable timbre preservation and context-adaptive expression, we propose a hierarchical variational conditioning module with an utterance-level speaker encoder to better balance stable timbre preservation and context-adaptive expression. This enables timbre reuse while keeping expression adaptive to the current utterance and, in dialogue, the surrounding context. We also build a comprehensive evaluation protocol for both single-utterance and dialogue settings. Experiments show that CapTalk achieves state-of-the-art performance on a single-utterance voice design benchmark and delivers better expression controllability and contextual appropriateness in multi-turn dialogue. Audio samples are available at: https://anonymous.4open.science/api/repo/Captalk-D601/file/index.html.

Paper Structure

This paper contains 39 sections, 16 equations, 2 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Architecture
  4. Overview
  5. Single-Utterance and Dialogue Modeling
  6. Single-Utterance Modeling
  7. Dialogue Modeling
  8. Overall Objective
  9. Factorized Hierarchical Variational Timbre Conditioning
  10. Task Formulation, Data Construction, and Evaluation Protocol
  11. Data Sources and Caption Construction
  12. CoT Design and Attribute Extraction
  13. Evaluation Protocol and Benchmark Analysis
  14. Experiments
  15. Comparative Experiments
  16. ...and 24 more sections

Figures (2)

  • Figure 1: Overview of CapTalk. (a) Hierarchical variational timbre conditioning. The bottom part shows the unified caption-conditioned text-audio autoregressive framework for both single-utterance and dialogue voice design.
  • Figure 2: Overview of data construction and evaluation. (A) Qwen3-Omni-based caption annotation for public and internal speech data. (B) Single-utterance and dialogue data construction with dialogue CoT extraction. (C) Evaluation for single-utterance and dialogue voice design.