Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GeoMVD: Geometry-Enhanced Multi-View Generation Model Based on Geometric Information Extraction

Jiaqi Wu, Yaosen Chen, Shuyuan Zhu

TL;DR

GeoMVD addresses the challenge of generating coherent multi-view images from a single input by introducing a geometry-guided diffusion framework. It leverages depth, normal maps, and foreground segmentation to create a BiNI proxy surface, then conditions diffusion with tri-modal features and a decoupled geometry-enhanced attention mechanism. An adaptive geometry intensity modulation strategy, including a multi-view feature mask and a stepwise scheduling of geometry attention, improves cross-view consistency while preserving detail. On the GSO and out-of-domain datasets, GeoMVD outperforms state-of-the-art baselines in cross-view coherence and visual fidelity.

Abstract

Multi-view image generation holds significant application value in computer vision, particularly in domains like 3D reconstruction, virtual reality, and augmented reality. Most existing methods, which rely on extending single images, face notable computational challenges in maintaining cross-view consistency and generating high-resolution outputs. To address these issues, we propose the Geometry-guided Multi-View Diffusion Model, which incorporates mechanisms for extracting multi-view geometric information and adjusting the intensity of geometric features to generate images that are both consistent across views and rich in detail. Specifically, we design a multi-view geometry information extraction module that leverages depth maps, normal maps, and foreground segmentation masks to construct a shared geometric structure, ensuring shape and structural consistency across different views. To enhance consistency and detail restoration during generation, we develop a decoupled geometry-enhanced attention mechanism that strengthens feature focus on key geometric details, thereby improving overall image quality and detail preservation. Furthermore, we apply an adaptive learning strategy that fine-tunes the model to better capture spatial relationships and visual coherence between the generated views, ensuring realistic results. Our model also incorporates an iterative refinement process that progressively improves the output quality through multiple stages of image generation. Finally, a dynamic geometry information intensity adjustment mechanism is proposed to adaptively regulate the influence of geometric data, optimizing overall quality while ensuring the naturalness of generated images. More details can be found on the project page: https://sobeymil.github.io/GeoMVD.com.

GeoMVD: Geometry-Enhanced Multi-View Generation Model Based on Geometric Information Extraction

TL;DR

GeoMVD addresses the challenge of generating coherent multi-view images from a single input by introducing a geometry-guided diffusion framework. It leverages depth, normal maps, and foreground segmentation to create a BiNI proxy surface, then conditions diffusion with tri-modal features and a decoupled geometry-enhanced attention mechanism. An adaptive geometry intensity modulation strategy, including a multi-view feature mask and a stepwise scheduling of geometry attention, improves cross-view consistency while preserving detail. On the GSO and out-of-domain datasets, GeoMVD outperforms state-of-the-art baselines in cross-view coherence and visual fidelity.

Abstract

Multi-view image generation holds significant application value in computer vision, particularly in domains like 3D reconstruction, virtual reality, and augmented reality. Most existing methods, which rely on extending single images, face notable computational challenges in maintaining cross-view consistency and generating high-resolution outputs. To address these issues, we propose the Geometry-guided Multi-View Diffusion Model, which incorporates mechanisms for extracting multi-view geometric information and adjusting the intensity of geometric features to generate images that are both consistent across views and rich in detail. Specifically, we design a multi-view geometry information extraction module that leverages depth maps, normal maps, and foreground segmentation masks to construct a shared geometric structure, ensuring shape and structural consistency across different views. To enhance consistency and detail restoration during generation, we develop a decoupled geometry-enhanced attention mechanism that strengthens feature focus on key geometric details, thereby improving overall image quality and detail preservation. Furthermore, we apply an adaptive learning strategy that fine-tunes the model to better capture spatial relationships and visual coherence between the generated views, ensuring realistic results. Our model also incorporates an iterative refinement process that progressively improves the output quality through multiple stages of image generation. Finally, a dynamic geometry information intensity adjustment mechanism is proposed to adaptively regulate the influence of geometric data, optimizing overall quality while ensuring the naturalness of generated images. More details can be found on the project page: https://sobeymil.github.io/GeoMVD.com.

Paper Structure

This paper contains 31 sections, 2 equations, 12 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Text-to-Image Diffusion Models.
  4. Multiview Generation Based on T2I Models.
  5. Method
  6. Multi-view Geometric Information Extraction Module
  7. Conditional Feature Extraction Process
  8. Decoupled Geometry-Enhanced Attention Mechanism
  9. Geometric Information Intensity Modulation Mechanism
  10. Multi-view Geometric Feature Mask:
  11. Dynamic Geometric Attention Strength Scheduling:
  12. Experiment
  13. Experimental Results
  14. Qualitative Comparison
  15. Quantitative Comparison
  16. ...and 16 more sections

Figures (12)

  • Figure 1: GeoMVD performs excellently on single-image–driven multi-view three-dimensional generation, is compatible with real photographs, synthesized images, and two-dimensional illustrations, and maintains cross-view consistency and high quality.
  • Figure 2: The process of the multi-view diffusion model guided by geometric information.
  • Figure 3: Schematic Diagram of the Multi-view Geometric Information Extraction Module.
  • Figure 4: Decoupled Geometry-Enhanced Attention Mechanism
  • Figure 5: Variation of Geometric Attention Scaling Factor with Diffusion Steps
  • ...and 7 more figures