AID: Adapting Image2Video Diffusion Models for Instruction-guided Video Prediction

TL;DR

This work tackles text-guided video prediction by transferring a large, pre-trained image-to-video diffusion model (Stable Video Diffusion) to instruction-driven tasks. It introduces a multimodal prompting pipeline using a Multimodal Large Language Model (LLM Llava) to predict future video states and a Dual Query Transformer (DQFormer) to fuse textual and visual conditions into a unified Multi-Condition (MCondition) for conditioning the diffusion model. To enable efficient domain adaptation with limited data, three adapters (spatial, short-term temporal, long-term temporal) are added while freezing the backbone, allowing fast transfer to SSv2, Bridge Data, and EpicKitchens-100. Empirical results show substantial improvements (often >50% in FVD) over prior TVP methods, with qualitative demonstrations highlighting improved frame consistency and instruction fidelity.

Abstract

Text-guided video prediction (TVP) involves predicting the motion of future frames from the initial frame according to an instruction, which has wide applications in virtual reality, robotics, and content creation. Previous TVP methods make significant breakthroughs by adapting Stable Diffusion for this task. However, they struggle with frame consistency and temporal stability primarily due to the limited scale of video datasets. We observe that pretrained Image2Video diffusion models possess good priors for video dynamics but they lack textual control. Hence, transferring Image2Video models to leverage their video dynamic priors while injecting instruction control to generate controllable videos is both a meaningful and challenging task. To achieve this, we introduce the Multi-Modal Large Language Model (MLLM) to predict future video states based on initial frames and text instructions. More specifically, we design a dual query transformer (DQFormer) architecture, which integrates the instructions and frames into the conditional embeddings for future frame prediction. Additionally, we develop Long-Short Term Temporal Adapters and Spatial Adapters that can quickly transfer general video diffusion models to specific scenarios with minimal training costs. Experimental results show that our method significantly outperforms state-of-the-art techniques on four datasets: Something Something V2, Epic Kitchen-100, Bridge Data, and UCF-101. Notably, AID achieves 91.2% and 55.5% FVD improvements on Bridge and SSv2 respectively, demonstrating its effectiveness in various domains. More examples can be found at our website https://chenhsing.github.io/AID.

Figures (11)

• Figure 1: (a) SVD is an Image2Video generative model that duplicates the initial image and combines this with noises to generate uncontrollable videos through a diffusion model. (b) Our method uses the initial first $K$ frames of the video and a mask of subsequent frames as conditions. Text conditions are also injected into the model to guide the prediction of future frames.
• Figure 2: (a) The pipeline includes a 3D U-Net for diffusion and a DQFormer for text conditioning. (b) The parameters of the original 3D U-Net are frozen, we only fine-tune the parameters of the newly added adapter during training.
• Figure 3: We input the initial frame of the video along with the text instruction of the video to be predicted into a multimodal large language model, allowing it to predict multiple state stages of the temporal changes based on the image and text.
• Figure 4: For TVP tasks, we design the DQFormer architecture that integrates multiple conditions. The initial frame and two textual prompts are processed by CLIP clip encoder to extract features. The upper branch of DQFormer aligns the visual feature with the textual instruction feature. Meanwhile, the lower branch decomposes the global prompt features into frame-level features. Finally, the MCondition is integrated into each frame of the video through cross-attention.
• Figure 5: The overview of three adapters.