Social Caption: Evaluating Social Understanding in Multimodal Models

TL;DR

Social Caption introduces a multidimensional framework for evaluating social understanding in multimodal large language models, extending beyond traditional QA by defining Social Inference (SI), Holistic Social Analysis (HSA), and Directed Social Analysis (DSA). It combines human and MLLM judge evaluations on the Social IQ 2.0-based dataset, revealing that spoken context and architectural choices substantially impact SI, while HSA and DSA capture richer, more nuanced aspects of social interaction. The study shows open-source models can approach or match large closed-source systems on SI and sometimes exceed them on HSA/DSA, and demonstrates the potential of MLLMs as scalable judges for social understanding, despite biases and alignment challenges. The work provides actionable insights into design trade-offs, evaluation rubrics, and ethical considerations, advocating community adoption and further research into longer, more diverse social video data.

Abstract

Social understanding abilities are crucial for multimodal large language models (MLLMs) to interpret human social interactions. We introduce Social Caption, a framework grounded in interaction theory to evaluate social understanding abilities of MLLMs along three dimensions: Social Inference (SI), the ability to make accurate inferences about interactions; Holistic Social Analysis (HSA), the ability to generate comprehensive descriptions of interactions; Directed Social Analysis (DSA), the ability to extract relevant social information from interactions. We analyze factors influencing model performance in social understanding, such as scale, architectural design, and spoken context. Experiments with MLLM judges contribute insights about scaling automated evaluation of multimodal social understanding.

Figures (11)

• Figure 1: Social Caption, a framework that evaluates social understanding of MLLMs, going beyond social inference to evaluate holistic social analysis and directed social analysis abilities.
• Figure 2: Social inference performance across model sizes, with and without spoken context. Higher scores indicate stronger social inference capabilities.
• Figure 3: HSA and DSA performance of models across their six evaluation sub-dimensions, visualized as heat maps. Each cell shows the mean human evaluation score (1–5 scale) for a model–dimension pair, where higher values indicate stronger performance. Color intensity corresponds to score magnitude.
• Figure 4: Qualitative example of MiniCPM-V2.6 responses evaluated by MLLM judges. The top section shows a representative video frame, the question, and the SI responses from both MiniCPM-V2.6 and MLLM judges. The middle section presents the DSA response of MiniCPM-V2.6, including detailed information on relevant scene details, individuals, interactions and context. The bottom section visualizes human and MLLM judge evaluations of the DSA response in a radar plot, accompanied by judges’ comments explaining reasoning.
• Figure 5: Prompt template to evaluate the SI dimension without providing transcriptions.