Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fuse3D: Generating 3D Assets Controlled by Multi-Image Fusion

Xuancheng Jin, Rengan Xie, Wenting Zheng, Rui Wang, Hujun Bao, Yuchi Huo

TL;DR

The proposed Fuse3D is the first method capable of controllable 3D asset generation from multiple condition images and can flexibly fuse multiple 2D image regions into coherent 3D structures, resulting in high-quality 3D assets.

Abstract

Recently, generating 3D assets with the control of condition images has achieved impressive quality. However, existing 3D generation methods are limited to handling a single control objective and lack the ability to utilize multiple images to independently control different regions of a 3D asset, which hinders their flexibility in applications. We propose Fuse3D, a novel method that enables generating 3D assets under the control of multiple images, allowing for the seamless fusion of multi-level regional controls from global views to intricate local details. First, we introduce a Multi-Condition Fusion Module to integrate the visual features from multiple image regions. Then, we propose a method to automatically align user-selected 2D image regions with their associated 3D regions based on semantic cues. Finally, to resolve control conflicts and enhance local control features from multi-condition images, we introduce a Local Attention Enhancement Strategy that flexibly balances region-specific feature fusion. Overall, we introduce the first method capable of controllable 3D asset generation from multiple condition images. The experimental results indicate that Fuse3D can flexibly fuse multiple 2D image regions into coherent 3D structures, resulting in high-quality 3D assets. Code and data for this paper are at https://jinnmnm.github.io/Fuse3d.github.io/.

Fuse3D: Generating 3D Assets Controlled by Multi-Image Fusion

TL;DR

The proposed Fuse3D is the first method capable of controllable 3D asset generation from multiple condition images and can flexibly fuse multiple 2D image regions into coherent 3D structures, resulting in high-quality 3D assets.

Abstract

Recently, generating 3D assets with the control of condition images has achieved impressive quality. However, existing 3D generation methods are limited to handling a single control objective and lack the ability to utilize multiple images to independently control different regions of a 3D asset, which hinders their flexibility in applications. We propose Fuse3D, a novel method that enables generating 3D assets under the control of multiple images, allowing for the seamless fusion of multi-level regional controls from global views to intricate local details. First, we introduce a Multi-Condition Fusion Module to integrate the visual features from multiple image regions. Then, we propose a method to automatically align user-selected 2D image regions with their associated 3D regions based on semantic cues. Finally, to resolve control conflicts and enhance local control features from multi-condition images, we introduce a Local Attention Enhancement Strategy that flexibly balances region-specific feature fusion. Overall, we introduce the first method capable of controllable 3D asset generation from multiple condition images. The experimental results indicate that Fuse3D can flexibly fuse multiple 2D image regions into coherent 3D structures, resulting in high-quality 3D assets. Code and data for this paper are at https://jinnmnm.github.io/Fuse3d.github.io/.
Paper Structure (54 sections, 7 equations, 24 figures, 5 tables)

This paper contains 54 sections, 7 equations, 24 figures, 5 tables.

Table of Contents

  1. Introduction
  2. related work
  3. 2D image Generation
  4. 3D Generative Models
  5. Controllable 3D Generation
  6. method
  7. Preliminaries and Latent Initialization
  8. TRELLIS
  9. Voxel initialization from a global image.
  10. Multi-condition Fusion Module
  11. 3D Semantic-Aware Alignment
  12. Forward Alignment: Image-Guided Voxel Selection.
  13. Reverse Alignment: Voxel-Guided Image Patch Selection.
  14. Local Attention Enhancement
  15. experiment
  16. ...and 39 more sections

Figures (24)

  • Figure 1: Overview. Given multiple condition images, we first utilize a Multi-Condition Fusion Module (MCFM) to obtain fused unified condition tokens. Subsequently, a 3D Semantic-Aware Alignment Module is used to establish the correspondence between the condition tokens and their target 3D structures. Finally, a Local Attention Enhancement Module is introduced to generate the target fused 3D asset.
  • Figure 2: The processing pipeline of MCFM to fuse multiple regions.
  • Figure 3: The processing pipeline of 3D Semantic-Aware Alignment to to identify the 2D-to-3D correspondences.
  • Figure 4: The pipeline of local attention enhancement module.
  • Figure 5: Generation quality comparison with previous methods.
  • ...and 19 more figures