Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GeCo: A Differentiable Geometric Consistency Metric for Video Generation

Leslie Gu, Junhwa Hur, Charles Herrmann, Fangneng Zhan, Todd Zickler, Deqing Sun, Hanspeter Pfister

TL;DR

GeCo presents a differentiable, geometry-grounded metric that fuses dense motion residuals with depth-based structure cues to detect geometric deformation and occlusion-inconsistency artifacts in videos generated from static scenes. It yields per-pixel inconsistency maps, enabling precise localization of artifacts and backpropagation for inference-time guidance. The authors introduce WarpBench and OccluBench to validate deformation and occlusion artifacts, and GeCo-Eval to benchmark multiple T2V models across varied scenarios. They demonstrate that GeCo-guided inference improves geometric consistency without fine-tuning and reveal common failure modes, such as spurious object motion, that the guidance can mitigate. Overall, GeCo advances both evaluation and control of geometry in video generation, highlighting geometry-aware approaches as essential for robust 3D-consistent video synthesis.

Abstract

We introduce GeCo, a geometry-grounded metric for jointly detecting geometric deformation and occlusion-inconsistency artifacts in static scenes. By fusing residual motion and depth priors, GeCo produces interpretable, dense consistency maps that reveal these artifacts. We use GeCo to systematically benchmark recent video generation models, uncovering common failure modes, and further employ it as a training-free guidance loss to reduce deformation artifacts during video generation.

GeCo: A Differentiable Geometric Consistency Metric for Video Generation

TL;DR

GeCo presents a differentiable, geometry-grounded metric that fuses dense motion residuals with depth-based structure cues to detect geometric deformation and occlusion-inconsistency artifacts in videos generated from static scenes. It yields per-pixel inconsistency maps, enabling precise localization of artifacts and backpropagation for inference-time guidance. The authors introduce WarpBench and OccluBench to validate deformation and occlusion artifacts, and GeCo-Eval to benchmark multiple T2V models across varied scenarios. They demonstrate that GeCo-guided inference improves geometric consistency without fine-tuning and reveal common failure modes, such as spurious object motion, that the guidance can mitigate. Overall, GeCo advances both evaluation and control of geometry in video generation, highlighting geometry-aware approaches as essential for robust 3D-consistent video synthesis.

Abstract

We introduce GeCo, a geometry-grounded metric for jointly detecting geometric deformation and occlusion-inconsistency artifacts in static scenes. By fusing residual motion and depth priors, GeCo produces interpretable, dense consistency maps that reveal these artifacts. We use GeCo to systematically benchmark recent video generation models, uncovering common failure modes, and further employ it as a training-free guidance loss to reduce deformation artifacts during video generation.
Paper Structure (41 sections, 19 equations, 12 figures, 6 tables)

This paper contains 41 sections, 19 equations, 12 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. GeCo: A Differentiable Geometric Consistency Metric for Video Generation
  4. Metric for Geometric Consistency
  5. Training-free Guidance with GeCo
  6. Benchmark Datasets for GeCo Validation
  7. WarpBench: Synthetic Deformation Dataset
  8. OccluBench: Occlusion Inconsistency Dataset
  9. Experiments
  10. GeCo-Eval: Benchmark Suite for T2V Models
  11. GeCo‑Guided Inference
  12. Findings on Video Generation Models
  13. Conclusion
  14. More Qualitative Results
  15. Interpretability Comparison
  16. ...and 26 more sections

Figures (12)

  • Figure 1: Geometric deformation detection on a generated video.Top: Input frames; the white box marks the target frame for detection. Middle: Zoomed-in deformations. Red box: the front chess piece (indicated by the arrow) gradually moves toward the piece behind it until they merge into a single piece, with the merged region highlighted by a red dashed circle. Blue box: a bishop morphs into a queen. Bottom: Comparison of inconsistency maps. MEt3R Asim2025MEt3RMM produces diffuse errors and fails to localize geometric shifts. WorldScore duan2025worldscore captures inconsistencies but does not accurately localize surface deformations. Motion cue highlights non-rigid motion artifacts, but remains undefined in occlusion regions and thus misses occlusion inconsistency artifacts. GeCo fuses motion and depth cues, accurately pinpointing the artifacts while retaining coverage over visibility-inconsistent regions, yielding a sharp and interpretable inconsistency map.
  • Figure 2: GeCo pipeline. Within a sliding window, we jointly estimate dense optical flow and 3D geometry (depth and camera pose) for frame pairs. We compute residual motion and depth errors and fuse them into scale-invariant inconsistency maps. Aggregation over the window localizes artifacts in the target frame, while motion and structure maps provide complementary diagnostics.
  • Figure 3: WarpBench deformation process.(Left) Input frame with foreground segmentation mask (cyan), sampled thin-plate spline (TPS) control points (red), and their destination points (blue). (Middle) Warped frame after the TPS deformation. (Right) Ground-truth dense displacement field from the deformation.
  • Figure 4: OccluBench. An example sequence where a region in the image center is (i) visible and empty, (ii) occluded, and (iii) re-revealed with a new object, forming a controlled sudden-appearance artifact.
  • Figure 5: GeCo guidance improves geometric consistency for 3D reconstruction. Top: 3D reconstructions from videos generated by CogVideoX‑5B without (left) and with GeCo guidance (right). Bottom: corresponding video frames. Both guided videos yield cleaner geometry with fewer deformation and drifting artifacts across views, which enables higher reconstruction quality.
  • ...and 7 more figures