Slot-ID: Identity-Preserving Video Generation from Reference Videos via Slot-Based Temporal Identity Encoding

TL;DR

Slot-ID tackles identity preservation in text-to-video generation by conditioning a frozen diffusion-transformer backbone on a short reference clip. It introduces a Sinkhorn-routed slot-based Temporal ID Encoder that outputs $S$ identity tokens capturing motion dynamics, enabling robust appearance maintenance across pose, illumination, and expression changes without per-identity fine-tuning. Across extensive experiments and a human study, Slot-ID outperforms strong baselines in Face Similarity, Naturalness, and Text Alignment, while maintaining prompt fidelity and visual realism. The method delivers a lightweight, plug-in conditioning mechanism suitable for personalized media, previsualization, and streaming where reliable identity retention and dynamic fidelity are crucial.

Abstract

Producing prompt-faithful videos that preserve a user-specified identity remains challenging: models need to extrapolate facial dynamics from sparse reference while balancing the tension between identity preservation and motion naturalness. Conditioning on a single image completely ignores the temporal signature, which leads to pose-locked motions, unnatural warping, and "average" faces when viewpoints and expressions change. To this end, we introduce an identity-conditioned variant of a diffusion-transformer video generator which uses a short reference video rather than a single portrait. Our key idea is to incorporate the dynamics in the reference. A short clip reveals subject-specific patterns, e.g., how smiles form, across poses and lighting. From this clip, a Sinkhorn-routed encoder learns compact identity tokens that capture characteristic dynamics while remaining pretrained backbone-compatible. Despite adding only lightweight conditioning, the approach consistently improves identity retention under large pose changes and expressive facial behavior, while maintaining prompt faithfulness and visual realism across diverse subjects and prompts.

Figures (6)

• Figure 1: Identity-preserving text-to-video generation with Slot-ID. From a reference video (left), we compute a slot-based temporal identity code to condition generation. The synthesized clips (right) follow diverse instructions—changing actions, scenes, and apparel—while consistently preserving the subject’s identity; colored text highlights key attributes.
• Figure 2: Failures from single-image references.(a, c) Reference portraits. (b)Face deformation: view changes warp facial geometry (stretched cheeks/jawline, eye misalignment).
• Figure 3: Pipeline overview. A text prompt, a background-neutral face reference, and a reference video are encoded to provide conditioning signals for generation. A Sinkhorn-routed slot reader then iteratively refines learnable slot queries: (1) compute query–token similarity scores; (2) apply temperature-scaled Sinkhorn normalization to obtain a (near) doubly-stochastic transport matrix; (3) aggregate values; (4) update queries with a GRU to yield identity slots.
• Figure 4: Identity-preserving video generation. Top: reference video (3 frames; leftmost used by image-conditioned baselines) and prompt; rows: each method with face crops. Baselines exhibit identity drift, expression/lip misalignment, over-smoothing, and flicker under non-frontal poses and background motion; ours preserves facial structure and expression/lip coherence, yielding sharper, temporally stable frames with consistent backgrounds. Best viewed zoomed in.
• Figure 5: Large-motion stress test. Baselines often hide the face or show identity drift/deformation; ours performs well.