Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MV-S2V: Multi-View Subject-Consistent Video Generation

Ziyang Song, Xinyu Gong, Bangya Liu, Zelin Zhao

TL;DR

MV-S2V tackles multi-view subject-to-video generation by conditioning a diffusion-based video model on multiple reference views to enforce 3D subject consistency. It introduces a synthetic data curation pipeline and a Temporally Shifted RoPE (TS-RoPE) conditioning mechanism, enabling effective separation of cross-subject and cross-view references. The approach uses a shared 3D VAE, DiT denoiser, and Rectified Flow training with classifier-free guidance to produce high-fidelity videos. Evaluations on OC and HOI scenarios show superior multi-view and 3D consistency compared with single-view baselines and prior methods, highlighting its potential for advertising and AR applications.

Abstract

Existing Subject-to-Video Generation (S2V) methods have achieved high-fidelity and subject-consistent video generation, yet remain constrained to single-view subject references. This limitation renders the S2V task reducible to an S2I + I2V pipeline, failing to exploit the full potential of video subject control. In this work, we propose and address the challenging Multi-View S2V (MV-S2V) task, which synthesizes videos from multiple reference views to enforce 3D-level subject consistency. Regarding the scarcity of training data, we first develop a synthetic data curation pipeline to generate highly customized synthetic data, complemented by a small-scale real-world captured dataset to boost the training of MV-S2V. Another key issue lies in the potential confusion between cross-subject and cross-view references in conditional generation. To overcome this, we further introduce Temporally Shifted RoPE (TS-RoPE) to distinguish between different subjects and distinct views of the same subject in reference conditioning. Our framework achieves superior 3D subject consistency w.r.t. multi-view reference images and high-quality visual outputs, establishing a new meaningful direction for subject-driven video generation. Our project page is available at: https://szy-young.github.io/mv-s2v

MV-S2V: Multi-View Subject-Consistent Video Generation

TL;DR

MV-S2V tackles multi-view subject-to-video generation by conditioning a diffusion-based video model on multiple reference views to enforce 3D subject consistency. It introduces a synthetic data curation pipeline and a Temporally Shifted RoPE (TS-RoPE) conditioning mechanism, enabling effective separation of cross-subject and cross-view references. The approach uses a shared 3D VAE, DiT denoiser, and Rectified Flow training with classifier-free guidance to produce high-fidelity videos. Evaluations on OC and HOI scenarios show superior multi-view and 3D consistency compared with single-view baselines and prior methods, highlighting its potential for advertising and AR applications.

Abstract

Existing Subject-to-Video Generation (S2V) methods have achieved high-fidelity and subject-consistent video generation, yet remain constrained to single-view subject references. This limitation renders the S2V task reducible to an S2I + I2V pipeline, failing to exploit the full potential of video subject control. In this work, we propose and address the challenging Multi-View S2V (MV-S2V) task, which synthesizes videos from multiple reference views to enforce 3D-level subject consistency. Regarding the scarcity of training data, we first develop a synthetic data curation pipeline to generate highly customized synthetic data, complemented by a small-scale real-world captured dataset to boost the training of MV-S2V. Another key issue lies in the potential confusion between cross-subject and cross-view references in conditional generation. To overcome this, we further introduce Temporally Shifted RoPE (TS-RoPE) to distinguish between different subjects and distinct views of the same subject in reference conditioning. Our framework achieves superior 3D subject consistency w.r.t. multi-view reference images and high-quality visual outputs, establishing a new meaningful direction for subject-driven video generation. Our project page is available at: https://szy-young.github.io/mv-s2v
Paper Structure (22 sections, 10 equations, 8 figures, 3 tables)

This paper contains 22 sections, 10 equations, 8 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Video Foundation Models
  4. Subject-Consistent Image Generation
  5. Subject-Consistent Video Generation
  6. Dataset Construction
  7. Synthetic Data Curation
  8. Real-world Data Capture
  9. MV-S2V
  10. Preliminary: T2V Base Model
  11. Multi-View Reference Conditioning
  12. Training and Inference
  13. Experiments
  14. Benchmark
  15. Evaluation Metrics
  16. ...and 7 more sections

Figures (8)

  • Figure 1: Synthetic data curation pipeline for MV-S2V, where the use of existing I2V models enables highly customized training data generation. Video captioning and data filtering stages are omitted for brevity.
  • Figure 2: Illustration about our MV-S2V framework along with different designs for multi-view reference conditioning.
  • Figure 3: Illustration about our multi-view / 3D subject consistency metrics.
  • Figure 4: Qualitative results of ablation study for reference conditioning. Artifacts in generated results, i.e., object deformation, abrupt changes, are highlighted.
  • Figure 5: Qualitative results of all methods on Object-Centric (OC) scenes. Inconsistencies and artifacts in generated results are highlighted.
  • ...and 3 more figures