AICL: Action In-Context Learning for Video Diffusion Model

TL;DR

The paper tackles open-domain video generation where action understanding is limited by training data. It proposes AICL, a framework that leverages in-context learning from reference videos via an Action Prism to distill motion features and an Action Integration mechanism that injects this motion through a new temporal cross-attention path while freezing the original diffusion backbone. Results show state-of-the-art improvements across three baseline diffusion models on five metrics and demonstrate strong generalization to unseen actions and to image-to-video tasks, with minimal inference overhead. This approach enables more diverse, motion-consistent action generation without requiring fine-tuning, advancing practical open-domain video synthesis.

Abstract

The open-domain video generation models are constrained by the scale of the training video datasets, and some less common actions still cannot be generated. Some researchers explore video editing methods and achieve action generation by editing the spatial information of the same action video. However, this method mechanically generates identical actions without understanding, which does not align with the characteristics of open-domain scenarios. In this paper, we propose AICL, which empowers the generative model with the ability to understand action information in reference videos, similar to how humans do, through in-context learning. Extensive experiments demonstrate that AICL effectively captures the action and achieves state-of-the-art generation performance across three typical video diffusion models on five metrics when using randomly selected categories from non-training datasets.

Figures (9)

• Figure 1: Some video generation models struggle to generate certain actions. For example, the original model cannot understand and generate "salsa". The video edit methods only directly copy motion from reference videos. The model with AICL (Action In-Context Learning) can comprehend relevant videos without training on this specific action with a few videos as references.
• Figure 2: Illustration of the proposed AICL for text-to-video diffusion models. AICL contains two parts: 1) Action Prism, marked with pink shading, to extract action features, and 2) Action Integration, marked with purple shading, to integrate action features. S and T stand for spatial and temporal respectively.
• Figure 3: Qualitative results of comparison between baseline models and AICL with single reference video.
• Figure 4: The visual of generated video feature by PCA.
• Figure 5: An example of wrong action reference video.