EgoSocial: Benchmarking Proactive Intervention Ability of Omnimodal LLMs via Egocentric Social Interaction Perception

TL;DR

This work tackles the problem of proactive intervention by omnimodal LLMs in egocentric social settings. It introduces EgoSocial, a large-scale egocentric benchmark with 13,500 video-question pairs and eight social-cue annotations, and EgoSoD, a thinking-graph-based method that fuses audio-visual cues to decide when to intervene. The authors show that current OLMMs struggle to determine precise intervention timing, but EgoSoD significantly improves timing and overall social interaction detection across multiple models, highlighting the value of structured multimodal cues and graph-based reasoning. The dataset and method aim to shift research toward intelligent, minimally disruptive AI assistance in AR/VR environments, with broader implications for socially aware AI agents.

Abstract

As AR/VR technologies become integral to daily life, there's a growing need for AI that understands human social dynamics from an egocentric perspective. However, current LLMs often lack the social awareness to discern when to intervene as AI assistant. This leads to constant, socially unaware responses that may disrupt natural conversation and negatively impact user focus. To address these limitations, we introduce EgoSocial, a large-scale egocentric dataset with 13,500 social video-question pairs, specifically designed to benchmark intervention in social interaction perception. We also present an in-depth analysis of current omnimodal LLMs (OLLMs) to assess their effectiveness in detecting diverse social contextual cues. Experiments show that OLLMs still struggle to detect the intervention timing (14.4% for Gemini 2.5 Pro). We also propose EgoSoD (EgoSocial Detection), an end-to-end method for robustly discerning social dynamics. Informed by our OLLM analysis, EgoSoD integrates multimodal contextual cues (e.g., audio and visual cues) into a social thinking graph, dynamically modeling participants and interactions. Our method proactively detects intervention timing and social interactions, precisely determining when to intervene. Our EgoSoD improves Phi-4 by 45.6% and Gemini 2.5 Pro by 9.9% on Intervention Timing performance, and improves Phi-4 by 20.4% and Gemini 2.5 Pro by 6.9% on overall Social Interaction performance. We will release the dataset and code soon.

Figures (5)

• Figure 1: Task Overview. We define 8 social interaction cues from coarse environmental cues to fine-grained personal attentional state cues covering both audio and vision. Our EgoSoD helps LLMs make intelligent timing decisions for intervention.
• Figure 2: Data distribution and correlation on our EgoSocial dataset.
• Figure 3: EgoSoD Overview. We propose a thinking-graph-based method that captures rich contextual features to guide OLLMs in making intelligent timing decisions for intervention.
• Figure 4: Demo of how EgoSoD helps OLLMs to detect social interaction. In this case, there are not that much valid visual information as there is neither looking at the wearer nor the wearer is looking at someone. So our EgoSoD guide the model to focus more on the audio part, which effectively outputs the right answer.
• Figure 5: Data distribution and correlation on our EgoSocial dataset.