Scaling Zero-Shot Reference-to-Video Generation

TL;DR

<3-5 sentence high-level summary> Saber tackles the scalability challenge of reference-to-video generation by proposing a zero-shot framework trained exclusively on video-text pairs. It introduces masked frames as dynamic references, a mask-augmented training regime, and a tailored attention mechanism to learn identity-consistent, reference-aware representations without explicit R2V data. The approach generalizes to varying numbers of references and outperforms R2V-data-trained methods on OpenS2V-Eval, demonstrating strong subject fidelity and cross-modal alignment. Emergent abilities include multi-view integration and robust cross-modal alignment, making Saber a scalable, flexible solution for personalized video generation with minimal data curation.

Abstract

Reference-to-video (R2V) generation aims to synthesize videos that align with a text prompt while preserving the subject identity from reference images. However, current R2V methods are hindered by the reliance on explicit reference image-video-text triplets, whose construction is highly expensive and difficult to scale. We bypass this bottleneck by introducing Saber, a scalable zero-shot framework that requires no explicit R2V data. Trained exclusively on video-text pairs, Saber employs a masked training strategy and a tailored attention-based model design to learn identity-consistent and reference-aware representations. Mask augmentation techniques are further integrated to mitigate copy-paste artifacts common in reference-to-video generation. Moreover, Saber demonstrates remarkable generalization capabilities across a varying number of references and achieves superior performance on the OpenS2V-Eval benchmark compared to methods trained with R2V data.

Figures (8)

• Figure 1: Saber is a zero-shot reference-to-video method trained only on video-text pairs. It preserves identity and appearance while coherently integrating single/multiple references into videos guided by text prompts.
• Figure 2: Masked reference generation. Given a video, the mask generator produces diverse random masks, which are then applied to each randomly sampled video frame with mask augmentation.
• Figure 3: Model design overview. Masked frames serve as reference images and are concatenated to the video tokens in latent space. Self-attention enables interaction between video and reference tokens under the attention mask, while cross-attention incorporates text guidance for semantic alignment. The VAE, text encoder, and timestep components are omitted for clarity.
• Figure 4: Qualitative comparison with existing R2V methods. We compare Saber with Kling1.6 kling, Phantom liu2025phantom, and VACE jiang2025vace across four scenarios: single/multiple human and object references. Saber accurately preserves subject identity and appearance, integrates multiple references coherently, and generates smoother, more visually consistent videos.
• Figure 5: Effect of mask augmentation. Without mask augmentation, the model shows copy-paste artifacts by directly copying reference content. Applying augmentation enables more natural and coherent video generation.