Video2LoRA: Unified Semantic-Controlled Video Generation via Per-Reference-Video LoRA

TL;DR

Video2LoRA is proposed, a scalable and generalizable framework for semantic-controlled video generation that conditions on a reference video that achieves coherent, semantically aligned generation across diverse conditions and exhibits strong zero-shot generalization to unseen semantics.

Abstract

Achieving semantic alignment across diverse video generation conditions remains a significant challenge. Methods that rely on explicit structural guidance often enforce rigid spatial constraints that limit semantic flexibility, whereas models tailored for individual control types lack interoperability and adaptability. These design bottlenecks hinder progress toward flexible and efficient semantic video generation. To address this, we propose Video2LoRA, a scalable and generalizable framework for semantic-controlled video generation that conditions on a reference video. Video2LoRA employs a lightweight hypernetwork to predict personalized LoRA weights for each semantic input, which are combined with auxiliary matrices to form adaptive LoRA modules integrated into a frozen diffusion backbone. This design enables the model to generate videos consistent with the reference semantics while preserving key style and content variations, eliminating the need for any per-condition training. Notably, the final model weights less than 150MB, making it highly efficient for storage and deployment. Video2LoRA achieves coherent, semantically aligned generation across diverse conditions and exhibits strong zero-shot generalization to unseen semantics.

Figures (4)

• Figure 1: Video2LoRA is a unified framework for semantic-controllable video generation. It takes a reference video containing the desired semantics as input and employs a HyperNetwork to generate lightweight, semantic-specific LoRA modules. By integrating these adaptive components into a frozen video diffusion backbone, Video2LoRA achieves high-quality video generation in both within-domain and out-of-domain scenarios.
• Figure 2: Overview of the proposed Video2LoRA framework. A reference semantic video is first fed into the HyperNetwork, where a 3D-VAE encoder extracts spatio-temporal latent features that are linearly projected into the layer-wise LightLoRA subspaces. The projected features are concatenated with zero-initialized weight tokens and processed by a Transformer decoder, which iteratively predicts the LightLoRA components $(A_{\text{pred}}, B_{\text{pred}})$ for each diffusion layer. These predicted components are then fused with the trainable auxiliary matrices $(A_{\text{aux}}, B_{\text{aux}})$ to form the final semantic-specific LoRA weights. The resulting LoRA adapters are injected into the frozen DiT backbone and optimized end-to-end with the vanilla diffusion loss, enabling semantic-controllable video generation from reference videos.
• Figure 3: Qualitative comparison with VFXCreator liu2025vfx and Ominieffect mao2025omni on the OpenVFX dataset. CogVideoX* refers to the CogVideoX model after supervised fine-tuning on our dataset.
• Figure 4: Out-of-Domain Comparison