Towards Online Multi-Modal Social Interaction Understanding

TL;DR

This work tackles the challenge of real-time multimodal social interaction understanding by formalizing Online-MMSI, where systems must respond using only historical dialogue and video. It introduces Online-MMSI-VLM, a framework that combines (i) multi-party conversation forecasting to enrich linguistic context via a coarse-to-fine prediction of upcoming turns and utterances, and (ii) social-aware visual prompting to highlight gaze, gestures, and group dynamics in annotated frames. The model is trained with two objectives, $\mathcal{L}_{\text{mmsi}}$ and $\mathcal{L}_{\text{forecast}}$, optimized jointly as $\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{mmsi}} + \mathcal{L}_{\text{forecast}}$, and leverages parameter-efficient fine-tuning (LoRA) on state-of-the-art MLLMs. Extensive experiments on YouTube and Ego4D Werewolf datasets show consistent improvements over online baselines, with forecasting and visual prompting providing additive gains across speaking target identification, pronoun coreference resolution, and mentioned player prediction, demonstrating strong potential for practical, real-time social AI systems.

Abstract

Multimodal social interaction understanding (MMSI) is critical in human-robot interaction systems. In real-world scenarios, AI agents are required to provide real-time feedback. However, existing models often depend on both past and future contexts, which hinders them from applying to real-world problems. To bridge this gap, we propose an online MMSI setting, where the model must resolve MMSI tasks using only historical information, such as recorded dialogues and video streams. To address the challenges of missing the useful future context, we develop a novel framework, named Online-MMSI-VLM, that leverages two complementary strategies: multi-party conversation forecasting and social-aware visual prompting with multi-modal large language models. First, to enrich linguistic context, the multi-party conversation forecasting simulates potential future utterances in a coarse-to-fine manner, anticipating upcoming speaker turns and then generating fine-grained conversational details. Second, to effectively incorporate visual social cues like gaze and gesture, social-aware visual prompting highlights the social dynamics in video with bounding boxes and body keypoints for each person and frame. Extensive experiments on three tasks and two datasets demonstrate that our method achieves state-of-the-art performance and significantly outperforms baseline models, indicating its effectiveness on Online-MMSI. The code and pre-trained models will be publicly released at: https://github.com/Sampson-Lee/OnlineMMSI.

Figures (6)

• Figure 1: Existing multimodal social interaction methods rely on both past and future contexts for accurate social understanding. This mechanism is not practical in real-world use case, which requires immediate interpretation and response. Our proposed model is able to correctly learn the features from solely the historical data, and respond to users in an online stream.
• Figure 2: The training pipeline in our Online-MMSI-VLM. The model takes the user prompt, historical dialogues and recorded video as input and generates an immediate response. To capture rich linguistic-visual social dynamics, we introduce two novel techniques: (i) multi-party conversation forecasting to enrich language context, and (ii) social-aware visual prompting to facilitate the social visual cues.
• Figure 3: Illustration of the challenges in transitioning from offline to online settings. In the online-MMSI scenario, the model lacks access to future utterances, which often contain crucial social cues.
• Figure 4: Qualitative results demonstrating the effectiveness of Online-MMSI-VLM for three online social tasks.
• Figure 5: Samples of generated and ground truth conversations.