Storynizor: Consistent Story Generation via Inter-Frame Synchronized and Shuffled ID Injection

TL;DR

Experimental results indicate that Storynizor demonstrates superior coherent story generation with high-fidelity character consistency, flexible postures, and vivid backgrounds compared to other character-specific methods.

Abstract

Recent advances in text-to-image diffusion models have spurred significant interest in continuous story image generation. In this paper, we introduce Storynizor, a model capable of generating coherent stories with strong inter-frame character consistency, effective foreground-background separation, and diverse pose variation. The core innovation of Storynizor lies in its key modules: ID-Synchronizer and ID-Injector. The ID-Synchronizer employs an auto-mask self-attention module and a mask perceptual loss across inter-frame images to improve the consistency of character generation, vividly representing their postures and backgrounds. The ID-Injector utilize a Shuffling Reference Strategy (SRS) to integrate ID features into specific locations, enhancing ID-based consistent character generation. Additionally, to facilitate the training of Storynizor, we have curated a novel dataset called StoryDB comprising 100, 000 images. This dataset contains single and multiple-character sets in diverse environments, layouts, and gestures with detailed descriptions. Experimental results indicate that Storynizor demonstrates superior coherent story generation with high-fidelity character consistency, flexible postures, and vivid backgrounds compared to other character-specific methods.

Figures (12)

• Figure 1: Comparison of Storynizor with existing methods. Storynizor shows superior performance when implemented in the original SD-base checkpoint in text-image alignment and inter-frame consistency.
• Figure 2: Overview of our proposed (a) Storynizor. Storynizor mainly contains two modules, ID-Injector and ID-Synchronizer. ID-Injector extracts ID features of reference characters with a Shuffling Reference Strategy (SRS), while ID-Synchronizer introduces a mask perceptual loss to modify cross-attention masks and utilizes an auto-mask self-attention module to ensure consistent generation of main characters across inter-frames, as well as vivid background.
• Figure 3: Cross attention map of each character during training. As the number of training steps increases, character attention maps gradually converge to accuracy within the constraints of mask perceptual loss.
• Figure 4: The structure of ID-Injector. The reference ID images are shuffled through Shuffling Reference Strategy(SPS), enhancing the pose flexibility across frames. A Resampler and several inter-frame controllers are introduced to integrate reference ID images into the ID-Synchronizer.
• Figure 5: StoryDB Visualization and Data processing pipeline.