Evolving Storytelling: Benchmarks and Methods for New Character Customization with Diffusion Models

TL;DR

Diffusion-based visual storytelling models struggle to generalize to unseen characters without leaking character information into training data. To address this, the authors introduce the NewEpisode benchmark for testing new-character adaptability and EpicEvo, a method that uses adversarial character alignment and distillation to customize a model from a single story. Empirical results show EpicEvo yields lower FID and higher CLIP-I/CLIP-T than baselines on NewEpisode Flintstones and Pororo, with ablations confirming the importance of both alignment and distillation. This work enables end-user creation of serialized cartoons featuring new characters with minimal data, advancing practical storytelling with diffusion models.

Abstract

Diffusion-based models for story visualization have shown promise in generating content-coherent images for storytelling tasks. However, how to effectively integrate new characters into existing narratives while maintaining character consistency remains an open problem, particularly with limited data. Two major limitations hinder the progress: (1) the absence of a suitable benchmark due to potential character leakage and inconsistent text labeling, and (2) the challenge of distinguishing between new and old characters, leading to ambiguous results. To address these challenges, we introduce the NewEpisode benchmark, comprising refined datasets designed to evaluate generative models' adaptability in generating new stories with fresh characters using just a single example story. The refined dataset involves refined text prompts and eliminates character leakage. Additionally, to mitigate the character confusion of generated results, we propose EpicEvo, a method that customizes a diffusion-based visual story generation model with a single story featuring the new characters seamlessly integrating them into established character dynamics. EpicEvo introduces a novel adversarial character alignment module to align the generated images progressively in the diffusive process, with exemplar images of new characters, while applying knowledge distillation to prevent forgetting of characters and background details. Our evaluation quantitatively demonstrates that EpicEvo outperforms existing baselines on the NewEpisode benchmark, and qualitative studies confirm its superior customization of visual story generation in diffusion models. In summary, EpicEvo provides an effective way to incorporate new characters using only one example story, unlocking new possibilities for applications such as serialized cartoons.

Figures (3)

• Figure 1: Illustration of the dataset construction. The proposed dataset has two main contributions compared with the original datasets (FlintStonesSV vlcstorygan and PororoSV storygan): first, as shown on the left side, the original text prompts lack description for supporting characters, making the training/evaluation less tractable while our dataset provides more comprehensive and consistent annotations. Second, the original data can not well establish the adapting performance of models on new characters since "the leakage of character information" is shown on the right side. We reorganize the dataset based on our previously more detailed annotated dataset so that there is no leakage of new characters in the customization set to the pertaining set.
• Figure 2: Illustration of EpicEvo. When generating the $i$-th image, the model takes all text inputs and previously generated images, encodes them using CLIP clip and BLIP blip text and visual encoders, and conditions the denoising network unet. To enable better story character customization, the denoising network has three training objectives: 1) predict the noise $\epsilon$ added to noisy latent $z_t$ at time step t such that the estimated noise $\epsilon_\theta$ is close to the ground truth noise $\epsilon$. This is reflected by the mean square loss (MSE) loss $\mathcal{L}_{mse}$; 2) maximizing the probability that the discriminator network will classify the generated image as a matching image w.r.t. to the reference image, i.e., minimize the adversarial character alignment loss $\mathcal{L}_{adv}^{G}$; 3) aligning with the pretrained model by minimizing a distillation loss $\mathcal{L}_{distill}$ calculated as the L2 distance between the noises estimated by the pretrained model and the customized model. We denote the latent denoised by the customized model and pretrained model as $\hat{z}_{t-1}^{\theta}$ and $\hat{z}_{t-1}^{\psi}$. Notably, both the diffusion process and adversarial alignment process operate in the latent space of the VAE vae network. Thus, for the adversarial alignment process, we estimate $\hat{z}_0$ instead of directly computing $\hat{z}_0$ as this is too computationally expensive. The dotted line here indicates that the estimated $\hat{z}_0$ can be decoded as a prediction of the generated image in the pixel space. We omit the complete process of adding noise to the image to obtain $z_t$ and the iterative nature of the reverse diffusion process for simplicity. Best viewed in color.
• Figure 3: Qualitative story character customization results from different baselines. For (a) and (b), we highlight the original visual story and highlight one of the generated frames containing the new character. For the rest of the image, we present sampled individual frames from stories to better demonstrate the effectiveness of EpicEvo under various conditions.