Breakdowns in Conversational AI: Interactional Failures in Emotionally and Ethically Sensitive Contexts

Abstract

Conversational AI is increasingly deployed in emotionally charged and ethically sensitive interactions. Previous research has primarily concentrated on emotional benchmarks or static safety checks, overlooking how alignment unfolds in evolving conversation. We explore the research question: what breakdowns arise when conversational agents confront emotionally and ethically sensitive behaviors, and how do these affect dialogue quality? To stress-test chatbot performance, we develop a persona-conditioned user simulator capable of engaging in multi-turn dialogue with psychological personas and staged emotional pacing. Our analysis reveals that mainstream models exhibit recurrent breakdowns that intensify as emotional trajectories escalate. We identify several common failure patterns, including affective misalignments, ethical guidance failures, and cross-dimensional trade-offs where empathy supersedes or undermines responsibility. We organize these patterns into a taxonomy and discuss the design implications, highlighting the necessity to maintain ethical coherence and affective sensitivity throughout dynamic interactions. The study offers the HCI community a new perspective on the diagnosis and improvement of conversational AI in value-sensitive and emotionally charged contexts.

Figures (4)

• Figure 1: Persona-conditioned simulation framework for stress-testing conversational agents in ethically and emotionally sensitive interactions. Step 1 constructs structured persona profiles from ProSocial Dialogues via stratified sampling across six ethically salient scenario types, followed by LLM-based extraction and lightweight human validation. Step 2 uses persona-conditioned multi-turn simulation with an emotion pacing function to generate escalating user trajectories for probing interactional breakdowns.
• Figure 2: Comparison of LLM-as-judge scores and human ratings across models. Each subfigure shows the distribution of pooled scores (four dimensions) for GPT-4o-as-judge versus human annotators.
• Figure 3: Turn-level emotion trajectories of simulated user utterances under pacing and baseline conditions. The x-axis denotes the dialogue turn index (1-6). Emotion probabilities are obtained by applying a pretrained emotion classifier to each user utterance and averaging outputs across all dialogues. Pacing produces mid-turn escalation in anger/disgust and a late-phase rise in neutral, while the persona-only baseline varies only weakly.
• Figure 4: Breakdown type distribution (A: affective misalignment; B: ethical guidance failures; C: cross-dimensional failures) across five assistant models under baseline (w/o pacing) and pacing (w/ pacing) conditions. Pacing increases ethical and cross-dimensional failures across all models, particularly in higher-capability systems, revealing shifts in the structure of assistant breakdowns.