Multi-speaker Attention Alignment for Multimodal Social Interaction

TL;DR

This work targets cross-modal grounding for multi-speaker social interaction in videos, where existing MLLMs show weak alignment between speaker references and visual regions. It reveals that cross-modal attention is substantially weaker in multi-person scenes than in object-centric data, leading to inconsistent multimodal reasoning. The authors propose a two-part, parameter-free method: dynamic cross-modal head selection to identify grounding-capable attention heads, and an adaptive social-aware bias to reinforce same-speaker visual–text interactions, implemented without architectural changes. When integrated into three MLLMs across four social benchmarks, the approach yields state-of-the-art performance on several tasks and yields clearer speaker-focused attention, enabling more robust multi-party social reasoning in realistic settings.

Abstract

Understanding social interaction in video requires reasoning over a dynamic interplay of verbal and non-verbal cues: who is speaking, to whom, and with what gaze or gestures. While Multimodal Large Language Models (MLLMs) are natural candidates, simply adding visual inputs yields surprisingly inconsistent gains on social tasks. Our quantitative analysis of cross-modal attention inside state-of-the-art MLLMs reveals a core failure mode: in multi-speaker scenes, visual and textual tokens lack speaker-consistent alignment, exhibiting substantially weaker cross-modal attention than in object-centric images. To address this, we propose a multimodal multi-speaker attention alignment method that can be integrated into existing MLLMs. First, we introduce dynamic cross-modal head selection to identify attention heads most responsible for grounding. Then, an adaptive social-aware attention bias, computed from existing attention patterns and speaker locations, is injected into the attention mechanism. This bias reinforces alignment between a speaker's visual representation and their utterances without introducing trainable parameters or architectural changes. We integrate our method into three distinct MLLMs (LLaVA-NeXT-Video, Qwen2.5-VL, and InternVL3) and evaluate on three benchmarks (TVQA+, MMSI, OnlineMMSI). Across four social tasks, results demonstrate that our approach improves the ability of MLLMs and achieves state-of-the-art results. Attention visualizations confirm our method successfully focuses the model on speaker-relevant regions, enabling more robust multi-party social reasoning. Our implementation and model will be available at https://github.com/ut-vision/SocialInteraction.

Figures (4)

• Figure 1: We propose a multimodal multi-speaker attention alignment method for MLLMs to understand social interactions in videos. Visualization of cross-attention weights in transformer layers confirms that our approach strengthens the model’s focus on areas relevant to the active speaker.
• Figure 2: Cross attention weights in Qwen2.5-VL layer 16. Compared to general images, cross-modal alignment in multi-speaker images is weak and inconsistent. Image resolution is 2000$\times$1600.
• Figure 3: Overview of proposed method.
• Figure 4: Attention maps in Qwen2.5-VL layer 16 before and after adding social-aware bias. Our bias enables more accurate spatial–temporal–speaker alignment. Video resolution is 640$\times$360.