CoCoEmo: Composable and Controllable Human-Like Emotional TTS via Activation Steering

TL;DR

CoCoEmo addresses the challenge of compositional and misaligned emotional expression in hybrid TTS by introducing activation steering in a two-stage architecture comprising a text-to-speech language model (SLM) and a flow-based decoder. The authors demonstrate that emotional prosody is primarily encoded in the SLM, not the flow module, enabling reliable single- and mixed-emotion steering via inference-time injections of carefully constructed steering vectors. They present a discriminability-driven framework to identify productive steering sites (mid-to-late SLM layers and attention) and introduce a mean-difference based method to extract emotion steering vectors, including a weighted mixed-emotion formulation for compositional control. A novel multi-rater evaluation protocol quantifies mixed-emotion synthesis and text–emotion mismatch handling, showing that steering improves E-SIM, TEP, and correlation metrics across backbones (CosyVoice2 and IndexTTS2) and datasets, while preserving speaker similarity and intelligibility. The work offers a practical, plug-in approach for richer, human-like emotional speech with broad implications for assistive, educational, and human–computer interaction applications, alongside considerations for misuse prevention and ethical deployment.

Abstract

Emotional expression in human speech is nuanced and compositional, often involving multiple, sometimes conflicting, affective cues that may diverge from linguistic content. In contrast, most expressive text-to-speech systems enforce a single utterance-level emotion, collapsing affective diversity and suppressing mixed or text-emotion-misaligned expression. While activation steering via latent direction vectors offers a promising solution, it remains unclear whether emotion representations are linearly steerable in TTS, where steering should be applied within hybrid TTS architectures, and how such complex emotion behaviors should be evaluated. This paper presents the first systematic analysis of activation steering for emotional control in hybrid TTS models, introducing a quantitative, controllable steering framework, and multi-rater evaluation protocols that enable composable mixed-emotion synthesis and reliable text-emotion mismatch synthesis. Our results demonstrate, for the first time, that emotional prosody and expressive variability are primarily synthesized by the TTS language module instead of the flow-matching module, and also provide a lightweight steering approach for generating natural, human-like emotional speech.

Figures (14)

• Figure 1: Overview of our method.Left: Stage-1: The SLM generates speech tokens; steering vectors are injected at selected layers and operators. Stage-2: Flow-matching and vocoder produce the final waveform. Right: Steering vector construction.
• Figure 2: Energy contours under cross-conditioning. Top (SLM-driven) produce distinct prosodic patterns across emotions; Bottom (Flow-driven) yield largely overlapping contours.
• Figure 3: Emotion discriminability across layers and operations (CosyVoice2). Mid-to-late layers and attention show high separability.
• Figure 4: Mixed-emotion synthesis results for CosyVoice2 on IEMOCAP. Higher values indicate better performance.
• Figure 5: E-SIM of high text–emotion mismatch synthesis under low, mid, and high conditions using CosyVoice2 on IEMOCAP.