Manga109Dialog: A Large-scale Dialogue Dataset for Comics Speaker Detection

TL;DR

This work addresses the lack of large-scale, reliable speaker-text annotations in comics by introducing Manga109Dialog, the largest dataset of 132,692 speaker-to-text pairs built on Manga109 with explicit speaker bounding-box links. It proposes a deep learning framework based on scene graph generation (SGG) that detects character and text regions and predicts speaker-text relations, augmented by a frame-reading order mechanism to capture comics-specific cues. The authors establish a new evaluation metric Recall@(#text) and show that frame-aware SGG methods outperform rule-based baselines, achieving over 75% total accuracy on PredCls and demonstrating reasonable generalization to other comic styles (eBDtheque). The work provides a solid dataset and a strong baseline for future research, paving the way for NLP-enhanced, multimodal analysis of comics and applications such as automated translation and audiobook character assignment.

Abstract

The expanding market for e-comics has spurred interest in the development of automated methods to analyze comics. For further understanding of comics, an automated approach is needed to link text in comics to characters speaking the words. Comics speaker detection research has practical applications, such as automatic character assignment for audiobooks, automatic translation according to characters' personalities, and inference of character relationships and stories. To deal with the problem of insufficient speaker-to-text annotations, we created a new annotation dataset Manga109Dialog based on Manga109. Manga109Dialog is the world's largest comics speaker annotation dataset, containing 132,692 speaker-to-text pairs. We further divided our dataset into different levels by prediction difficulties to evaluate speaker detection methods more appropriately. Unlike existing methods mainly based on distances, we propose a deep learning-based method using scene graph generation models. Due to the unique features of comics, we enhance the performance of our proposed model by considering the frame reading order. We conducted experiments using Manga109Dialog and other datasets. Experimental results demonstrate that our scene-graph-based approach outperforms existing methods, achieving a prediction accuracy of over 75%.

Figures (28)

• Figure 1: An example of speaker detection. Courtesy of Akamatsu Ken.
• Figure 2: Annotation rules. Courtesy of Inokuma Shinobu.
• Figure 3: Easy.
• Figure 4: Hard.
• Figure 5: Total.