EmoShift: Lightweight Activation Steering for Enhanced Emotion-Aware Speech Synthesis

TL;DR

EmoShift tackles the challenge of precise, interpretable emotion control in TTS by introducing a lightweight activation-steering framework with an EmoSteer layer that learns emotion-specific latent offsets in the output embedding space. It uses an LLM-based TTS backbone and a plug-and-play steering mechanism, requiring only about 10M trainable parameters, and supports inference-time intensity control via a steering gain $\alpha$. In experiments on the English subset of ESD, EmoShift consistently outperforms zero-shot and fully fine-tuned baselines in both objective and subjective measures, while preserving naturalness and speaker similarity; enabling higher emotional intensity without changing the target emotion. The work demonstrates the practicality of activation steering for emotion control in TTS and suggests broader applicability to other controllable attributes and modalities.

Abstract

Achieving precise and controllable emotional expression is crucial for producing natural and context-appropriate speech in text-to-speech (TTS) synthesis. However, many emotion-aware TTS systems, including large language model (LLM)-based designs, rely on scaling fixed emotion embeddings or external guidance, limiting their ability to model emotion-specific latent characteristics. To address this gap, we present EmoShift, a lightweight activation-steering framework incorporating a EmoSteer layer, which learns a steering vector for each target emotion in the output embedding space to capture its latent offset and maintain stable, appropriate expression across utterances and categories. With only 10M trainable parameters,less than 1/30 of full fine-tuning, EmoShift outperforms zero-shot and fully fine-tuned baselines in objective and subjective evaluations, enhancing emotional expressiveness while preserving naturalness and speaker similarity. Further analysis confirms the proposed EmoSteer layer's effectiveness and reveals its potential for controllable emotional intensity in speech synthesis.

Figures (3)

• Figure 1: Illustration of steering vector operation in activation space.
• Figure 2: Overview of the proposed EmoShift framework. Ground-truth speech inputs $Y$ used only during training and omitted during inference.
• Figure 3: Overall emotion recognition accuracy with different steering factor $\alpha$ at inference.