A Concise Agent is Less Expert: Revealing Side Effects of Using Style Features on Conversational Agents

TL;DR

This paper investigates stylistic prompts for conversational agents and reveals that style features are deeply entangled, producing systematic side effects when a Main Feature is prompted. It combines a broad survey of $127$ ACL papers to identify $12$ common features, a controlled synthetic-dialogue study generating $12{,}200$ messages, and a pairwise win-rate evaluation framework to quantify cross-feature influences. The authors introduce CASSE, a dataset annotating side-feature interactions, and evaluate two mitigation approaches—Prompt Intervention and Steering Intervention—finding that while they can restore suppressed traits, they often degrade the primary target style, highlighting the need for multi-objective, principled style control. The work argues that faithful, isolated style control is insufficient for safe, targeted steering in LLMs and advocates for more sophisticated optimization or learning-based approaches. It provides a valuable resource (CASSE) for future research and calls for multi-objective methods to balance style, safety, and effectiveness in conversational agents.

Abstract

Style features such as friendly, helpful, or concise are widely used in prompts to steer the behavior of Large Language Model (LLM) conversational agents, yet their unintended side effects remain poorly understood. In this work, we present the first systematic study of cross-feature stylistic side effects. We conduct a comprehensive survey of 127 conversational agent papers from ACL Anthology and identify 12 frequently used style features. Using controlled, synthetic dialogues across task-oriented and open domain settings, we quantify how prompting for one style feature causally affects others via a pairwise LLM as a Judge evaluation framework. Our results reveal consistent and structured side effects, such as prompting for conciseness significantly reduces perceived expertise. They demonstrate that style features are deeply entangled rather than orthogonal. To support future research, we introduce CASSE (Conversational Agent Stylistic Side Effects), a dataset capturing these complex interactions. We further evaluate prompt based and activation steering based mitigation strategies and find that while they can partially restore suppressed traits, they often degrade the primary intended style. These findings challenge the assumption of faithful style control in LLMs and highlight the need for multi-objective and more principled approaches to safe, targeted stylistic steering in conversational agents.

Figures (11)

• Figure 1: Overall pipeline of our study. We first collect popular style features from our survey, and generate feature-guided messages in both task-oriented and daily dialog domain. Then we identify side effects of using style features, and finally mitigate these side effects via prompting and steering.
• Figure 2: Data collection pipeline for papers and style features selection of our survey.
• Figure 3: Style Feature Win Rate Matrix Across Three Models. The y-axis shows Main Features, and the x-axis shows Side Features for comparison. Each cell represents the average win rate of the styled response against the neutral response across three models. Red indicates a win rate $>$ 50% (positive alignment), while blue indicates a win rate $<$ 50% (negative impact). Average results per Domain are presented in Figure \ref{['fig:combined_heatmaps_averaeg_all']}.
• Figure 4: Performance comparison of prompting vs. steering Interventions. Top: Main Feature win rates. Bottom: Side Feature win rates.
• Figure 5: Extracted Feature Distribution. This figure shows the frequency of each unique style feature in the survey results, highlighting the most common style features such as helpful and empathetic.