Sentient Agent as a Judge: Evaluating Higher-Order Social Cognition in Large Language Models

TL;DR

This work introduces Sage, a fully automated evaluation framework that deploys a Sentient Agent to simulate evolving human emotions and inner thoughts during interactions, enabling the assessment of higher-order social cognition in LLMs. By embedding emotion estimation and goal-directed reasoning within dynamic, diverse evaluation environments, Sage provides a principled, scalable, and interpretable benchmark. The authors validate Sage through a 100-scenario supportive-dialogue benchmark and a public Sentient Leaderboard spanning 18 models, revealing strong correlations with human-centric measures (e.g., BLRI) and notable gaps between frontier systems and older baselines that conventional leaderboards miss. The framework also introduces concepts like token efficiency and a Social Cognition Coordinate to profile models, offering practical insights for developing more empathetic and socially adept language agents.

Abstract

Assessing how well a large language model (LLM) understands human, rather than merely text, remains an open challenge. To bridge the gap, we introduce Sentient Agent as a Judge (SAGE), an automated evaluation framework that measures an LLM's higher-order social cognition. SAGE instantiates a Sentient Agent that simulates human-like emotional changes and inner thoughts during interaction, providing a more realistic evaluation of the tested model in multi-turn conversations. At every turn, the agent reasons about (i) how its emotion changes, (ii) how it feels, and (iii) how it should reply, yielding a numerical emotion trajectory and interpretable inner thoughts. Experiments on 100 supportive-dialogue scenarios show that the final Sentient emotion score correlates strongly with Barrett-Lennard Relationship Inventory (BLRI) ratings and utterance-level empathy metrics, validating psychological fidelity. We also build a public Sentient Leaderboard covering 18 commercial and open-source models that uncovers substantial gaps (up to 4x) between frontier systems (GPT-4o-Latest, Gemini2.5-Pro) and earlier baselines, gaps not reflected in conventional leaderboards (e.g., Arena). SAGE thus provides a principled, scalable and interpretable tool for tracking progress toward genuinely empathetic and socially adept language agents.

Figures (12)

• Figure 1: (a) The rankings on our Sentient Leaderboard differ markedly from those of the conventional Arena Leaderboard, uncovering LLMs' ability to make people feel heard, not just answered. (b) The quadrant characterized by creativity (e.g., highly flexible interactions) and empathy (e.g., deep empathetic engagement) remains largely unoccupied, indicating that current LLMs still struggle to meet this demanding profile.
• Figure 2: An illustration of our proposed Sage, a novel framework to automatically assess higher-order social cognition in target LLMs.
• Figure 3: An illustration of the workflow of the Sentient Agent.
• Figure 4: Correlation between emotion and internal user thought. Overall correlation: 0.818.
• Figure 5: Correlation between emotion and dialogue utterance. Overall correlation: 0.788.