BridgeIV: Bridging Customized Image and Video Generation through Test-Time Autoregressive Identity Propagation

TL;DR

BridgeIV tackles the limited generalization and fidelity in customized text-to-video generation by introducing an autoregressive Structure and Texture Propagation Module (STPM) that transfers subject structure and texture across frames, coupled with a Test-Time Reward Optimization (TTRO) to refine fine-grained details. The method follows a three-stage pipeline: customization via DreamBooth-style CT2I adaptation, structure/texture injection with STPM, and latent enhancement through TTRO. Experiments on the VICO dataset show superior subject consistency and visual fidelity compared to zero-shot, two-stage, and tuning-based baselines, with quantitative gains in CLIP-I and DINO metrics. The work demonstrates that explicit, autoregressive propagation of structural and texture cues, augmented by test-time optimization, can significantly improve CT2V results and enable more reliable AI-driven storytelling visuals.

Abstract

Both zero-shot and tuning-based customized text-to-image (CT2I) generation have made significant progress for storytelling content creation. In contrast, research on customized text-to-video (CT2V) generation remains relatively limited. Existing zero-shot CT2V methods suffer from poor generalization, while another line of work directly combining tuning-based T2I models with temporal motion modules often leads to the loss of structural and texture information. To bridge this gap, we propose an autoregressive structure and texture propagation module (STPM), which extracts key structural and texture features from the reference subject and injects them autoregressively into each video frame to enhance consistency. Additionally, we introduce a test-time reward optimization (TTRO) method to further refine fine-grained details. Quantitative and qualitative experiments validate the effectiveness of STPM and TTRO, demonstrating improvements of 7.8 and 13.1 in CLIP-I and DINO consistency metrics over the baseline, respectively.

Figures (6)

• Figure 1: The first row presents the generated customized images, while the second row displays the video result. Although the customized image generation model achieves promising results, directly integrating a temporal motion module leads to structural and texture distortions in the video
• Figure 2: The processes of the structure and texture injection stage (the second stage) and the latent enhancement stage (the third stage). In the second stage, SPM extracts the reference subject's structure information $A^{*}_{ref}$ from the cross-attention of the CT2I model and propagates it to video frames through $\{\hat{A}_f\}$, using matching flows computed from the features, $\psi_{ref,t+1}$ and $\{\psi_{f,t+1}\}$ of the previous denoising step. Similarly, TPM utilizes the structure information $\{\hat{A}^*_{f}\}$ and matching flow to transfer the reference subject's texture information $V_{ref}$ into the video. In the third stage, two reward functions are designed in the latent and pixel domains, respectively, to correct the latent distribution, enhancing the consistency of the foreground.
• Figure 3: The visualization of normalized attention maps of the special token in different denoising steps. The highlighted regions in the attention maps correspond to the generated reference image (second column), indicating that the attention maps clearly capture the subject's structural information.
• Figure 4: The visualization of the warping results with the matching flows. The first column represents the generated reference image, its foreground mask, and the results after multiplying with the attention map. (a) The corresponding generated video frames. (b) The warping results from the previous frame. (c) The warped foreground masks. (d) The warping results multiplied with the attention maps. We can observe that the subject from the reference image or video frame is correctly warped to the appropriate position in the next frame, providing the subject's texture information.
• Figure 5: The comparisons with different state-of-the-art CT2V methods, including two-stages methods, VideoStudio long2024videostudio and the baseline TI-DB, tuning-based method, Dreamvideo wei2024dreamvideo, and zero-shot method, Videobooth jiang2024videobooth. It can be observed that the two-stage method is prone to structural deformation, while traditional tuning-based and zero-shot methods have weak representation capabilities for unique objects, resulting in poor consistency in the generated results.