Friends-MMC: A Dataset for Multi-modal Multi-party Conversation Understanding

TL;DR

This work defines multi-modal multi-party conversation (MMC) and introduces Friends-MMC, a dataset pairing 24k+ utterances with video context, speaker identities, face bounding boxes, and character names drawn from the TV show Friends. It formalizes two MMC tasks—conversation speaker identification and conversation response prediction—and presents a task-specific baseline that fuses visual (M1) and textual (M2) signals via a quadratic binary optimization solver, maximizing $f(X)=(1-\alpha)X^TAX+\alpha XB$ under $X\in\{0,1\}^{m\times l}$ and per-turn constraints. Experimental results show that visual cues are the primary drivers for speaker identification, with textual context providing a meaningful, complementary boost, while multi-modal pre-trained models underperform compared to the proposed baseline. For response prediction, incorporating speaker information consistently improves accuracy, with evidence that local context rather than global speaking style underpins the gains. The dataset and baseline code aim to catalyze speaker-aware modeling in MMC, enabling broader applications in embodied AI and real-world dialogues.

Abstract

Multi-modal multi-party conversation (MMC) is a less studied yet important topic of research due to that it well fits real-world scenarios and thus potentially has more widely-used applications. Compared with the traditional multi-modal conversations, MMC requires stronger character-centered understanding abilities as there are many interlocutors appearing in both the visual and textual context. To facilitate the study of this problem, we present Friends-MMC in this paper, an MMC dataset that contains 24,000+ unique utterances paired with video context. To explore the character-centered understanding of the dialogue, we also annotate the speaker of each utterance, the names and bounding bboxes of faces that appear in the video. Based on this Friends-MMC dataset, we further study two fundamental MMC tasks: conversation speaker identification and conversation response prediction, both of which have the multi-party nature with the video or image as visual context. For conversation speaker identification, we demonstrate the inefficiencies of existing methods such as pre-trained models, and propose a simple yet effective baseline method that leverages an optimization solver to utilize the context of two modalities to achieve better performance. For conversation response prediction, we fine-tune generative dialogue models on Friend-MMC, and analyze the benefits of speaker information. The code and dataset is publicly available at https://github.com/yellow-binary-tree/Friends-MMC and thus we call for more attention on modeling speaker information when understanding conversations.

Figures (4)

• Figure 1: An example of multi-modal multi-party conversation. The task of conversation speaker identification is to infer the dotted arrows pointing from characters to utterances, and the task of conversation response prediction is to infer the last utterance in the dotted rectangular. Only one frame of the video is shown as the visual context to avoid clutter.
• Figure 2: An overview of the construction process of Friends-MMC dataset.
• Figure 3: Model overview of the three modules in different colors: the visual ($M_1$) is yellow, the textual ($M_2$) is green, and the optimization solver taking vision and text reward matrix as input is blue.
• Figure 4: The change of accuracy with respect to $\alpha$. The dotted horizontal line shows the performance of only using the visual model.