DiffSensei: Bridging Multi-Modal LLMs and Diffusion Models for Customized Manga Generation

TL;DR

DiffSensei integrates diffusion-based image synthesis with a multimodal LLM as a text-compatible identity adapter to enable customized, multi-character manga generation with precise layout and dialogue control. The authors introduce MangaZero, a large-scale dataset of 43,264 manga pages and 427,147 panels, annotated for multi-character states and captions to support training and evaluation. Core innovations include masked cross-attention for per-character layout, a dialog layout embedding for bubbles, and an MLLM-based feature adapter that maps input character features to text-driven states, trained in a two-stage process. Quantitative and human evaluations show DiffSensei outperforms baselines on identity preservation, text alignment, and overall narrative quality, demonstrating a practical pathway to scalable, controllable manga generation. The work has potential impacts in manga production, education, and media by enabling rapid, personalized visual storytelling while emphasizing ethical data use through MangaZero.

Abstract

Story visualization, the task of creating visual narratives from textual descriptions, has seen progress with text-to-image generation models. However, these models often lack effective control over character appearances and interactions, particularly in multi-character scenes. To address these limitations, we propose a new task: \textbf{customized manga generation} and introduce \textbf{DiffSensei}, an innovative framework specifically designed for generating manga with dynamic multi-character control. DiffSensei integrates a diffusion-based image generator with a multimodal large language model (MLLM) that acts as a text-compatible identity adapter. Our approach employs masked cross-attention to seamlessly incorporate character features, enabling precise layout control without direct pixel transfer. Additionally, the MLLM-based adapter adjusts character features to align with panel-specific text cues, allowing flexible adjustments in character expressions, poses, and actions. We also introduce \textbf{MangaZero}, a large-scale dataset tailored to this task, containing 43,264 manga pages and 427,147 annotated panels, supporting the visualization of varied character interactions and movements across sequential frames. Extensive experiments demonstrate that DiffSensei outperforms existing models, marking a significant advancement in manga generation by enabling text-adaptable character customization. The project page is https://jianzongwu.github.io/projects/diffsensei/.

Figures (19)

• Figure 1: Results of DiffSensei. (a) Customized manga generation with controllable character images, panel captions, and layout conditions. Our DiffSensei successfully generates detailed character expressions and states following the panel captions. (b) Manga creation for real human images. The dialogues are post-edited by humans. The continuation is in the Appendix. We strongly recommend that the readers see the Appendix for more comprehensive results. Manga reading order: Right to left. Top to bottom.
• Figure 2: We construct MangaZero through three steps: 1) Download manga pages from the internet. 2) Annotate manga panels autonomously with pre-trained models. 3) Human calibration for the character ID annotation.
• Figure 3: The architecture of DiffSensei. In the first stage, we train a multi-character customized manga image generation model with layout control. The dialog embedding is added to the noised latent after the first convolution layer. All the parameters in the U-Net and feature extractor are trained. In the second stage, we finetune LoRA and resampler weights of an MLLM to adapt the source character features corresponding to the text prompt. We use the model in the first stage as the image generator and freeze its weights.
• Figure 4: Qualitative comparison with baselines. Baselines followed by a "*" use reference images as input rather than character images. Methods marked by "†" means re-trained with dialog embedding. Our model excels at preserving the characters while following the text prompt. Our DiffSensei successively generates highlighted details in panel captions. Better viewed with zoom-in.
• Figure 5: Human preference study on MangaZero eval set.