Bring Your Dreams to Life: Continual Text-to-Video Customization

TL;DR

CCVD introduces Continual Text-to-Video Customization (CTVC) to address learning new personalized subjects and motions without forgetting prior concepts and without overreliance on stored data. It couples a concept-specific attribute retention module with a task-aware aggregation strategy to preserve identity and selectively reuse past adapters. It also adds a controllable conditional synthesis module using layer-specific region attention and attention-guided noise estimation to prevent concept neglect in multi-concept videos. Experimental results on DreamVideo and Wan 2.1 backbones show state-of-the-art performance across single/multi-concept customization, style transfer, and video editing tasks.

Abstract

Customized text-to-video generation (CTVG) has recently witnessed great progress in generating tailored videos from user-specific text. However, most CTVG methods assume that personalized concepts remain static and do not expand incrementally over time. Additionally, they struggle with forgetting and concept neglect when continuously learning new concepts, including subjects and motions. To resolve the above challenges, we develop a novel Continual Customized Video Diffusion (CCVD) model, which can continuously learn new concepts to generate videos across various text-to-video generation tasks by tackling forgetting and concept neglect. To address catastrophic forgetting, we introduce a concept-specific attribute retention module and a task-aware concept aggregation strategy. They can capture the unique characteristics and identities of old concepts during training, while combining all subject and motion adapters of old concepts based on their relevance during testing. Besides, to tackle concept neglect, we develop a controllable conditional synthesis to enhance regional features and align video contexts with user conditions, by incorporating layer-specific region attention-guided noise estimation. Extensive experimental comparisons demonstrate that our CCVD outperforms existing CTVG baselines on both the DreamVideo and Wan 2.1 backbones. The code is available at https://github.com/JiahuaDong/CCVD.

Figures (15)

• Figure 1: Results of our model in continually learning new personalized concepts, including subjects and motions, under the CTVC setting. Compared with L2DM, our model achieves better performance on both the DreamVideo and Wan 2.1 backbones.
• Figure 2: CCVD architectural overview. It includes (a) a concept-specific attribute retention module, (b) a task-aware concept aggregation strategy to overcome catastrophic forgetting of previous concepts during training and testing, and (c) a controllable conditional synthesis with layer-specific region attention and attention-guided noise estimation to address concept neglect.
• Figure 3: Comparison results of single-concept video customization under the CTVC setting when the backbone is DreamVideo.
• Figure 4: Comparison results of multi-concept video customization under the CTVC setting when the backbone is DreamVideo.
• Figure 5: Comparisons of style transfer under CTVC setting when the backbone is DreamVideo.