Are they lovers or friends? Evaluating LLMs' Social Reasoning in English and Korean Dialogues

TL;DR

This work introduces SCRIPTS, a bilingual benchmark of about 1k dialogues (English and Korean) from movie scripts to evaluate LLMs' social relationship reasoning under uncertainty. It adopts a probabilistic labeling scheme (Highly Likely, Less Likely, Unlikely) and evaluates nine models, revealing substantial gaps in social reasoning, with English outperforming Korean and frequent Unlikely inferences across models. The study also tests chain-of-thought and thinking-enabled prompts, finding limited or context-dependent benefits, and highlights four failure modes related to address terms, cue aggregation, atypical relationships, and language/cultural features. By analyzing the impact of demographic and relational-information cues and demonstrating cross-linguistic differences, the paper argues for language- and culture-aware approaches to building socially aware LLMs with more robust and nuanced reasoning. The dataset and findings have practical implications for safer and more contextually appropriate human–AI interactions in diverse languages and cultures.

Abstract

As large language models (LLMs) are increasingly used in human-AI interactions, their social reasoning capabilities in interpersonal contexts are critical. We introduce SCRIPTS, a 1k-dialogue dataset in English and Korean, sourced from movie scripts. The task involves evaluating models' social reasoning capability to infer the interpersonal relationships (e.g., friends, sisters, lovers) between speakers in each dialogue. Each dialogue is annotated with probabilistic relational labels (Highly Likely, Less Likely, Unlikely) by native (or equivalent) Korean and English speakers from Korea and the U.S. Evaluating nine models on our task, current proprietary LLMs achieve around 75-80% on the English dataset, whereas their performance on Korean drops to 58-69%. More strikingly, models select Unlikely relationships in 10-25% of their responses. Furthermore, we find that thinking models and chain-of-thought prompting, effective for general reasoning, provide minimal benefits for social reasoning and occasionally amplify social biases. Our findings reveal significant limitations in current LLMs' social reasoning capabilities, highlighting the need for efforts to develop socially-aware language models.

Figures (6)

• Figure 1: Illustration of SCRIPTS and potential risks from failures in Social Relationship Reasoning.SCRIPTS provides three human-annotated relationship labels: Highly Likely, Less Likely, and Unlikely. In this example, Gemini-2.5-Flash incorrectly predicts a store clerk-customer relationship as a mother-child. Such misleading relational reasoning can lead to inappropriate responses and unsafe decisions, such as privacy leakage. Examples are from Korean dialogues, translated into English.
• Figure 2: Comparative Analysis of Relational Dimension Distributions for Six Relationship Types Present in Both English and Korean Top-10 Relations. Legend labels denote intimacy (O: intimate, X: unintimate, △: neutral), formality (O: formal, X: informal, △: neutral), and hierarchy (<: hierarchical, =: equal).
• Figure 3: Comparison of model performance in English and Korean datasets.Highly Likely represents the accuracy of the model’s majority response being a highly likely response, while Unlikely indicates the error rate where the model generate an unlikely response.
• Figure 4: Distribution of GPT-4o’s 30 failure cases by error type in English and Korean.
• Figure 5: Annotation Platform. The annotators can read the dialogue on the left side and annotate the relationships and relational dimensions on the right side.