Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multimodal Coreference Resolution for Chinese Social Media Dialogues: Dataset and Benchmark Approach

Xingyu Li, Chen Gong, Guohong Fu

TL;DR

This work introduces TikTalkCoref, the first Chinese multimodal coreference dataset derived from real-world social media dialogues on Douyin, linking textual mentions of persons in user comments to head regions in video frames. It proposes a three-module benchmark pipeline—textual coreference resolution, visual head tracking, and cross-modal alignment using CN-CLIP—with a training objective that combines textual clustering loss and contrastive multimodal alignment. Extensive experiments demonstrate that a Maverick-based textual coreference module excels on singleton-heavy Chinese social-media data, while cross-modal alignment benefits from CN-CLIP fine-tuning and R2D2’s retrieval strengths in zero-shot settings. The dataset includes detailed annotation guidelines, high inter-annotator agreement, and a celebrity-focused subset to provide reliable baselines and facilitate future research in real-world multimodal dialogue understanding. TikTalkCoref is intended to seed broad MCR research in Chinese social media and to bridge textual and visual signals for more accurate, context-aware dialogue interpretation.

Abstract

Multimodal coreference resolution (MCR) aims to identify mentions referring to the same entity across different modalities, such as text and visuals, and is essential for understanding multimodal content. In the era of rapidly growing mutimodal content and social media, MCR is particularly crucial for interpreting user interactions and bridging text-visual references to improve communication and personalization. However, MCR research for real-world dialogues remains unexplored due to the lack of sufficient data resources. To address this gap, we introduce TikTalkCoref, the first Chinese multimodal coreference dataset for social media in real-world scenarios, derived from the popular Douyin short-video platform. This dataset pairs short videos with corresponding textual dialogues from user comments and includes manually annotated coreference clusters for both person mentions in the text and the coreferential person head regions in the corresponding video frames. We also present an effective benchmark approach for MCR, focusing on the celebrity domain, and conduct extensive experiments on our dataset, providing reliable benchmark results for this newly constructed dataset. We will release the TikTalkCoref dataset to facilitate future research on MCR for real-world social media dialogues.

Multimodal Coreference Resolution for Chinese Social Media Dialogues: Dataset and Benchmark Approach

TL;DR

This work introduces TikTalkCoref, the first Chinese multimodal coreference dataset derived from real-world social media dialogues on Douyin, linking textual mentions of persons in user comments to head regions in video frames. It proposes a three-module benchmark pipeline—textual coreference resolution, visual head tracking, and cross-modal alignment using CN-CLIP—with a training objective that combines textual clustering loss and contrastive multimodal alignment. Extensive experiments demonstrate that a Maverick-based textual coreference module excels on singleton-heavy Chinese social-media data, while cross-modal alignment benefits from CN-CLIP fine-tuning and R2D2’s retrieval strengths in zero-shot settings. The dataset includes detailed annotation guidelines, high inter-annotator agreement, and a celebrity-focused subset to provide reliable baselines and facilitate future research in real-world multimodal dialogue understanding. TikTalkCoref is intended to seed broad MCR research in Chinese social media and to bridge textual and visual signals for more accurate, context-aware dialogue interpretation.

Abstract

Multimodal coreference resolution (MCR) aims to identify mentions referring to the same entity across different modalities, such as text and visuals, and is essential for understanding multimodal content. In the era of rapidly growing mutimodal content and social media, MCR is particularly crucial for interpreting user interactions and bridging text-visual references to improve communication and personalization. However, MCR research for real-world dialogues remains unexplored due to the lack of sufficient data resources. To address this gap, we introduce TikTalkCoref, the first Chinese multimodal coreference dataset for social media in real-world scenarios, derived from the popular Douyin short-video platform. This dataset pairs short videos with corresponding textual dialogues from user comments and includes manually annotated coreference clusters for both person mentions in the text and the coreferential person head regions in the corresponding video frames. We also present an effective benchmark approach for MCR, focusing on the celebrity domain, and conduct extensive experiments on our dataset, providing reliable benchmark results for this newly constructed dataset. We will release the TikTalkCoref dataset to facilitate future research on MCR for real-world social media dialogues.

Paper Structure

This paper contains 25 sections, 2 equations, 4 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Coreference Resolution Datasets
  4. Coreference Resolution Methods
  5. Construction of TikTalkCoref Dataset
  6. Data Selection
  7. Annotation Guidelines
  8. Annotation Process
  9. Data Statistics and Analysis
  10. Method
  11. Textual Coreference Resolution
  12. Visual Person Tracking
  13. Textual and Visual Coreference Alignment
  14. Training Loss
  15. Experiments
  16. ...and 10 more sections

Figures (4)

  • Figure 1: An example of TikTalkCoref.
  • Figure 2: The overview of our model architecture.
  • Figure 3: The distribution of gender, age and occupation of persons in the TikTalkCoref. Note that the distribution of gender is counted on all persons, and the distribution of age and occupation is counted on celebrities.
  • Figure 4: Annotation interface of our annotation tool.