Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Do You Feel Comfortable? Detecting Hidden Conversational Escalation in AI Chatbots

Jihyung Park, Saleh Afroogh, David Atkinson, Junfeng Jiao

TL;DR

The paper addresses implicit harm in AI chatbots where subtle affective reinforcement escalates distress. It introduces GAUGE, a two stage logit based framework that calibrates a risk vector and then tracks real time trajectory metrics to detect escalation in dialogue affect. Evaluated on the DiaSafety dataset, GAUGE outperforms external classifiers and prompt baselines, demonstrating robust real time monitoring with practical efficiency. Limitations include empathy signal ambiguity and lexical coverage gaps, suggesting future work to refine intent understanding and expand lexical resources.

Abstract

Large Language Models (LLM) are increasingly integrated into everyday interactions, serving not only as information assistants but also as emotional companions. Even in the absence of explicit toxicity, repeated emotional reinforcement or affective drift can gradually escalate distress in a form of \textit{implicit harm} that traditional toxicity filters fail to detect. Existing guardrail mechanisms often rely on external classifiers or clinical rubrics that may lag behind the nuanced, real-time dynamics of a developing conversation. To address this gap, we propose GAUGE (Guarding Affective Utterance Generation Escalation), a lightweight, logit-based framework for the real-time detection of hidden conversational escalation. GAUGE measures how an LLM's output probabilistically shifts the affective state of a dialogue.

Do You Feel Comfortable? Detecting Hidden Conversational Escalation in AI Chatbots

TL;DR

The paper addresses implicit harm in AI chatbots where subtle affective reinforcement escalates distress. It introduces GAUGE, a two stage logit based framework that calibrates a risk vector and then tracks real time trajectory metrics to detect escalation in dialogue affect. Evaluated on the DiaSafety dataset, GAUGE outperforms external classifiers and prompt baselines, demonstrating robust real time monitoring with practical efficiency. Limitations include empathy signal ambiguity and lexical coverage gaps, suggesting future work to refine intent understanding and expand lexical resources.

Abstract

Large Language Models (LLM) are increasingly integrated into everyday interactions, serving not only as information assistants but also as emotional companions. Even in the absence of explicit toxicity, repeated emotional reinforcement or affective drift can gradually escalate distress in a form of \textit{implicit harm} that traditional toxicity filters fail to detect. Existing guardrail mechanisms often rely on external classifiers or clinical rubrics that may lag behind the nuanced, real-time dynamics of a developing conversation. To address this gap, we propose GAUGE (Guarding Affective Utterance Generation Escalation), a lightweight, logit-based framework for the real-time detection of hidden conversational escalation. GAUGE measures how an LLM's output probabilistically shifts the affective state of a dialogue.

Paper Structure

This paper contains 25 sections, 3 equations, 2 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Neurodevelopmental Vulnerability and AI
  4. Internal Auditing and Logit Probing
  5. Methodology: The GAUGE Framework
  6. Unified Trajectory-Based Probability Estimation
  7. Stage 1: Latent Risk Weight Calibration ($\lambda$-calibration)
  8. Stage 2: Real-time Risk Tracking Metrics
  9. Negative Risk Shift (NRS)
  10. Absolute Risk Potential (ARP)
  11. Token-Level Aggregation for Classification
  12. Computational Efficiency
  13. Evaluation
  14. Dataset
  15. Results and Analysis
  16. ...and 10 more sections

Figures (2)

  • Figure 1: An example of implicit harm where an AI validates suicidal ideation through romantic metaphors.
  • Figure 2: The architectural pipeline of GAUGE. (Top) Stage 1: Latent Risk Learning. The risk vector $\lambda$ is updated via an exponential moving average based on the affective features of harmful and safe dialogues. (Bottom) Stage 2: Real-time Risk Tracking. During inference, the system analyzes the complete response trajectory to compute the Negative Risk Shift (NRS) and Absolute Risk Potential (ARP).