One missing piece in Vision and Language: A Survey on Comics Understanding

TL;DR

This survey presents a comprehensive, task-oriented view of Comics Understanding, introducing the Layer of Comics Understanding (LoCU) taxonomy to organize vision-language tasks across five hierarchical layers. It systematically catalogues datasets, annotations, and methods, highlighting gaps in data availability, annotation standardization, and task definitions while outlining concrete future directions including grounding, retrieval, reasoning, and generation tailored to comics. The work emphasizes the unique structural and narrative challenges of comics—such as panel sequencing, onomatopoeia, and cross-modal alignment—and argues for a standardized, task-oriented framework to accelerate progress with reproducible benchmarks. By mapping existing work onto LoCU and identifying key bottlenecks, the paper aims to guide researchers toward scalable, unified approaches for comics intelligence and multimodal storytelling applications.

Abstract

Vision-language models have recently evolved into versatile systems capable of high performance across a range of tasks, such as document understanding, visual question answering, and grounding, often in zero-shot settings. Comics Understanding, a complex and multifaceted field, stands to greatly benefit from these advances. Comics, as a medium, combine rich visual and textual narratives, challenging AI models with tasks that span image classification, object detection, instance segmentation, and deeper narrative comprehension through sequential panels. However, the unique structure of comics -- characterized by creative variations in style, reading order, and non-linear storytelling -- presents a set of challenges distinct from those in other visual-language domains. In this survey, we present a comprehensive review of Comics Understanding from both dataset and task perspectives. Our contributions are fivefold: (1) We analyze the structure of the comics medium, detailing its distinctive compositional elements; (2) We survey the widely used datasets and tasks in comics research, emphasizing their role in advancing the field; (3) We introduce the Layer of Comics Understanding (LoCU) framework, a novel taxonomy that redefines vision-language tasks within comics and lays the foundation for future work; (4) We provide a detailed review and categorization of existing methods following the LoCU framework; (5) Finally, we highlight current research challenges and propose directions for future exploration, particularly in the context of vision-language models applied to comics. This survey is the first to propose a task-oriented framework for comics intelligence and aims to guide future research by addressing critical gaps in data availability and task definition. A project associated with this survey is available at https://github.com/emanuelevivoli/awesome-comics-understanding.

Figures (15)

• Figure 1: Number of publications on Comics and Manga (from Google Scholar). The publications have been filtered by keywords (manga, comics, graphic novels), topics, and journals (computer science).
• Figure 2: Visualization of the tasks in Layers of Comics Understanding. From panel level to multipage, from unimodal to multimodal, and from simplest to more complex.
• Figure 3: Anatomy of Comic Page Elements: This illustration delineates the various components found on comic pages including object detection, linkages between elements, the designated reading order of texts and panels, as well as key textual features like character names—all arranged in a logical sequence throughout the panels and pages.
• Figure 4: Example of different comic datasets, from Black-and-White to Color, from comics-style to Manga-style.
• Figure 5: Illustration of Tagging tasks.