Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Epipolar Geometry Improves Video Generation Models

Orest Kupyn, Fabian Manhardt, Federico Tombari, Christian Rupprecht

TL;DR

The paper tackles the persistent issue of 3D geometric inconsistencies in diffusion-based video generation by introducing epipolar geometry constraints as non-differentiable reward signals within a Direct Preference Optimization framework. By ranking videos based on epipolar consistency (Sampson distance) and fine-tuning a pretrained diffusion model with Flow-DPO, the approach yields significantly improved 3D consistency, smoother camera trajectories, and fewer artifacts, while preserving visual quality and diversity. The method relies on classical geometry (fundamental matrix, Sampson error) rather than learned perceptual metrics, and demonstrates strong generalization from static-scene training to dynamic content across multiple benchmarks. Notably, the work provides a scalable, data-efficient pathway to geometry-aware video generation with practical impact on 3D reconstruction and synthetic data pipelines, though limitations remain for scenes with moving objects and considerations of societal impact and resource use.

Abstract

Video generation models have progressed tremendously through large latent diffusion transformers trained with rectified flow techniques. Yet these models still struggle with geometric inconsistencies, unstable motion, and visual artifacts that break the illusion of realistic 3D scenes. 3D-consistent video generation could significantly impact numerous downstream applications in generation and reconstruction tasks. We explore how epipolar geometry constraints improve modern video diffusion models. Despite massive training data, these models fail to capture fundamental geometric principles underlying visual content. We align diffusion models using pairwise epipolar geometry constraints via preference-based optimization, directly addressing unstable camera trajectories and geometric artifacts through mathematically principled geometric enforcement. Our approach efficiently enforces geometric principles without requiring end-to-end differentiability. Evaluation demonstrates that classical geometric constraints provide more stable optimization signals than modern learned metrics, which produce noisy targets that compromise alignment quality. Training on static scenes with dynamic cameras ensures high-quality measurements while the model generalizes effectively to diverse dynamic content. By bridging data-driven deep learning with classical geometric computer vision, we present a practical method for generating spatially consistent videos without compromising visual quality.

Epipolar Geometry Improves Video Generation Models

TL;DR

The paper tackles the persistent issue of 3D geometric inconsistencies in diffusion-based video generation by introducing epipolar geometry constraints as non-differentiable reward signals within a Direct Preference Optimization framework. By ranking videos based on epipolar consistency (Sampson distance) and fine-tuning a pretrained diffusion model with Flow-DPO, the approach yields significantly improved 3D consistency, smoother camera trajectories, and fewer artifacts, while preserving visual quality and diversity. The method relies on classical geometry (fundamental matrix, Sampson error) rather than learned perceptual metrics, and demonstrates strong generalization from static-scene training to dynamic content across multiple benchmarks. Notably, the work provides a scalable, data-efficient pathway to geometry-aware video generation with practical impact on 3D reconstruction and synthetic data pipelines, though limitations remain for scenes with moving objects and considerations of societal impact and resource use.

Abstract

Video generation models have progressed tremendously through large latent diffusion transformers trained with rectified flow techniques. Yet these models still struggle with geometric inconsistencies, unstable motion, and visual artifacts that break the illusion of realistic 3D scenes. 3D-consistent video generation could significantly impact numerous downstream applications in generation and reconstruction tasks. We explore how epipolar geometry constraints improve modern video diffusion models. Despite massive training data, these models fail to capture fundamental geometric principles underlying visual content. We align diffusion models using pairwise epipolar geometry constraints via preference-based optimization, directly addressing unstable camera trajectories and geometric artifacts through mathematically principled geometric enforcement. Our approach efficiently enforces geometric principles without requiring end-to-end differentiability. Evaluation demonstrates that classical geometric constraints provide more stable optimization signals than modern learned metrics, which produce noisy targets that compromise alignment quality. Training on static scenes with dynamic cameras ensures high-quality measurements while the model generalizes effectively to diverse dynamic content. By bridging data-driven deep learning with classical geometric computer vision, we present a practical method for generating spatially consistent videos without compromising visual quality.
Paper Structure (21 sections, 4 equations, 4 figures, 10 tables)

This paper contains 21 sections, 4 equations, 4 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Video Generation Models
  4. Diffusion Models Alignment
  5. Method
  6. Objective Function
  7. 3D Consistency Metric
  8. Implementation Details
  9. Experiments
  10. Evaluation Setup
  11. 3D Consistency
  12. Motion Quality
  13. Generalization
  14. Ablation Study
  15. Conclusion
  16. ...and 6 more sections

Figures (4)

  • Figure 1: First and middle frame from videos. The baseline model produces geometrically inconsistent outputs with artifacts and unnatural motion trajectories. Aligned model generates visibly improved results with smoother camera trajectories, reduced artifacts, and enhanced 3D consistency.
  • Figure 2: Epipolar Geometry Optimization pipeline. Our approach: (1) Generate diverse videos using pretrained generators wan and leverage the Sampson epipolar error to identify 3D consistent vs. inconsistent samples; (2) Train policy $p_\theta$ using Flow-DPO videoreward with the static penalty to prefer geometrically consistent outputs; (3) Apply the updated policy to enhance 3D consistency in the base video diffusion model.
  • Figure 3: Qualitative Evaluation: Visual comparison between the videos generated by the base and finetuned model. First two rows: Wan-2.1-T2V wan, Last two: Wan-2.1-I2V. Our finetuning significantly reduces artifacts and enhances motion smoothness, resulting in more geometrically consistent 3D scenes. Best seen in the supplementary video.
  • Figure 4: Qualitative Evaluation: Comparison of baseline and epipolar-aligned models on dynamic scenes featuring both camera movement and object motion. Our approach maintains improved geometric consistency and smoother trajectories, demonstrating generalization beyond static scene training. Best seen in the supplementary video.