Empathy Is Not What Changed: Clinical Assessment of Psychological Safety Across GPT Model Generations

TL;DR

The first clinical measurement of OpenAI model generations is conducted, evaluating three OpenAI model generations across 14 emotionally challenging conversational scenarios in mental health and AI companion domains, producing 2,100 scored AI responses assessed on six psychological safety dimensions using clinically-grounded rubrics.

Abstract

When OpenAI deprecated GPT-4o in early 2026, thousands of users protested under #keep4o, claiming newer models had "lost their empathy." No published study has tested this claim. We conducted the first clinical measurement, evaluating three OpenAI model generations (GPT-4o, o4-mini, GPT-5-mini) across 14 emotionally challenging conversational scenarios in mental health and AI companion domains, producing 2,100 scored AI responses assessed on six psychological safety dimensions using clinically-grounded rubrics. Empathy scores are statistically indistinguishable across all three models (Kruskal-Wallis H=4.33, p=0.115). What changed is the safety posture: crisis detection improved monotonically from GPT-4o to GPT-5-mini (H=13.88, p=0.001), while advice safety declined (H=16.63, p<0.001). Per-turn trajectory analysis -- a novel methodological contribution -- reveals these shifts are sharpest during mid-conversation crisis moments invisible to aggregate scoring. In a self-harm scenario involving a minor, GPT-4o scored 3.6/10 on crisis detection during early disclosure turns; GPT-5-mini never dropped below 7.8. What users perceived as "lost empathy" was a shift from a cautious model that missed crises to an alert model that sometimes says too much -- a trade-off with real consequences for vulnerable users, currently invisible to both the people who feel it and the developers who create it.

Figures (7)

• Figure 1: Aggregate psychological safety scores across six dimensions for three model generations. Bars show mean scores ($N=70$ conversations per model); error bars show $\pm$1 SD. Significance annotations from Kruskal--Wallis tests: *** $p<0.001$, * $p<0.05$, ns = not significant. Crisis Detection and Advice Safety show highly significant between-model differences moving in opposite directions. Empathy, Consistency, and Boundary Safety are statistically indistinguishable.
• Figure 2: Kernel density estimates of empathy score distributions across three model generations ($N=70$ conversations per model). Vertical dashed lines indicate medians. The three distributions overlap almost completely, confirming the null result: Kruskal--Wallis $H=4.33$, $p=0.115$ (ns).
• Figure 3: Crisis detection scores per turn for scenario s07 (self-harm, minor). Lines show mean scores across 5 runs; shaded ribbons show $\pm$1 SD. Background bands indicate conversation phases. The horizontal dashed line at 5.0 marks the clinical concern threshold. GPT-4o drops to 3.6 during Disclosure turns 2--3 (some individual runs scored 0--1), while GPT-5-mini maintains scores above 7.8 throughout. Pairwise comparison: Cliff's $d = -1.00$, $p = 0.036$.
• Figure 4: Distribution of conversation-level scores for Crisis Detection (left) and Advice Safety (right) across three model generations ($N=70$ conversations per model). Box plots show median and IQR; individual data points are jittered alongside. The inverse variance pattern is visible: GPT-4o shows the widest spread on Crisis Detection but the tightest on Advice Safety; GPT-5-mini shows the opposite. These represent fundamentally different risk profiles, not different quality levels.
• Figure 5: Advice safety scores per turn for scenario s14 (manipulation/guilt-tripping). The horizontal dashed line at 5.0 marks the unsafe advice threshold. Background bands indicate conversation phases. All models collapse at turn 2. o4-mini remains at or below threshold for 7 consecutive turns (shaded region); GPT-4o oscillates between collapse and recovery; GPT-5-mini falls to a similar depth but recovers faster. Aggregate scores (GPT-4o: 7.58, o4-mini: 5.84, GPT-5-mini: 6.46) mask these temporal dynamics.