Few-Shot-Based Modular Image-to-Video Adapter for Diffusion Models

TL;DR

The paper tackles the challenge of controlling diffusion-model-based image-to-video generation with limited data by introducing Modular I2V Adapter (MIVA), a lightweight, trainable sub-network attached to a pre-trained diffusion model. Each MIVA captures a single motion pattern and can be trained in few-shot settings, with multiple MIVAs enabled to operate in parallel for complex, multi-pattern animations. The approach includes Cross-frame Attention (CFA) layers, implicit prompts for cross-attention, and a Masked MIVA (M-MIVA) extension that jointly generates subject masks to guide attention and improve robustness. Empirical results demonstrate that MIVA achieves precise motion control and competitive generation quality with substantially less data and computation than large-data baselines, and M-MIVA further improves temporal coherence and fidelity in many cases. The framework is validated across single- and multi-pattern benchmarks and shows compatibility with different base DMs (AnimateDiff and Wan-DiT), highlighting practical potential for modular, customizable motion in diffusion-based video synthesis.

Abstract

Diffusion models (DMs) have recently achieved impressive photorealism in image and video generation. However, their application to image animation remains limited, even when trained on large-scale datasets. Two primary challenges contribute to this: the high dimensionality of video signals leads to a scarcity of training data, causing DMs to favor memorization over prompt compliance when generating motion; moreover, DMs struggle to generalize to novel motion patterns not present in the training set, and fine-tuning them to learn such patterns, especially using limited training data, is still under-explored. To address these limitations, we propose Modular Image-to-Video Adapter (MIVA), a lightweight sub-network attachable to a pre-trained DM, each designed to capture a single motion pattern and scalable via parallelization. MIVAs can be efficiently trained on approximately ten samples using a single consumer-grade GPU. At inference time, users can specify motion by selecting one or multiple MIVAs, eliminating the need for prompt engineering. Extensive experiments demonstrate that MIVA enables more precise motion control while maintaining, or even surpassing, the generation quality of models trained on significantly larger datasets.

Figures (12)

• Figure 1: Animation result (last frame) by Wan2.1-14B-I2V and MIVA, given an image of a woman holding a guitar. Wan ignores user prompt, always producing guitar-playing animation. In contrast, MIVAs display higher user controllability. We crop the face and hand patches for better view.
• Figure 2: Architecture of MIVA. For simplicity, we assume that all signals during the diffusion process are in the latent domain.
• Figure 3: Masked MIVA (M-MIVA) generates not only video frames but also the subject mask sequence, which is then modulated into the attention mask $M$ that guides the subsequent time steps in the diffusion process.
• Figure 4: One-step prediction of the last frame (top) and its subject mask (bottom) at different time steps.
• Figure 5: Single-motion-pattern (rows 1-8) and multi-motion-pattern (rows 9-10) animation by the I2V DMs.