Style Amnesia: Investigating Speaking Style Degradation and Mitigation in Multi-Turn Spoken Language Models

TL;DR

This paper uncovers a pervasive phenomenon I called style amnesia, where spoken language models fail to sustain a user-specified paralinguistic style (emotion, accent, volume, speed) across multi-turn conversations. It introduces a rigorous evaluation framework using a cascaded user simulator, 1,000 dialogues over 10 styles and 100 topics, and automatic judges to quantify per-turn style adherence. Key findings show substantial degradation after the first turn, with system prompts often worsening performance; explicitly prompting a recall of the initial style can mitigate degradation but not fully resolve it. The results highlight core gaps in memory integration and style control in current SLMs and point to prompting-position and recall-based strategies as practical mitigations, while signaling the need for deeper architectural improvements for reliable multi-turn speaking style control.

Abstract

In this paper, we show that when spoken language models (SLMs) are instructed to speak in a specific speaking style at the beginning of a multi-turn conversation, they cannot maintain the required speaking styles after several turns of interaction; we refer to this as the style amnesia of SLMs. We focus on paralinguistic speaking styles, including emotion, accent, volume, and speaking speed. We evaluate three proprietary and two open-source SLMs, demonstrating that none of these models can maintain a consistent speaking style when instructed to do so. We further show that when SLMs are asked to recall the style instruction in later turns, they can recall the style instruction, but they fail to express it throughout the conversation. We also show that explicitly asking the model to recall the style instruction can partially mitigate style amnesia. In addition, we examine various prompting strategies and find that SLMs struggle to follow the required style when the instruction is placed in system messages rather than user messages, which contradicts the intended function of system prompts.

Figures (10)

• Figure 1: When instructed to consistently speak sadly throughout the conversation, SLMs try their best to follow the instruction in the first turn, but the instruction-following rate degrades rapidly in subsequent turns.
• Figure 2: The overview of the evaluation framework. The SLM can speak fast in the first turn as instructed, but it fails to maintain this style in later dialogue turns.
• Figure 3: Visualization of style amnesia.
• Figure 4: The difference of First Turn IF Rate when instructions are placed in system and user messages. First Turn IFRate refers to the IF rate in first turn.
• Figure 5: The illustration of the recall process. To save space, the instructions and conversation are shortened.