Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

M3D: Dual-Stream Selective State Spaces and Depth-Driven Framework for High-Fidelity Single-View 3D Reconstruction

Luoxi Zhang, Pragyan Shrestha, Yu Zhou, Chun Xie, Itaru Kitahara

TL;DR

Experimental results indicate that the fusion of multi-scale features with depth information via the dual-branch feature extraction significantly boosts geometric consistency and fidelity, achieving state-of-the-art reconstruction performance.

Abstract

The precise reconstruction of 3D objects from a single RGB image in complex scenes presents a critical challenge in virtual reality, autonomous driving, and robotics. Existing neural implicit 3D representation methods face significant difficulties in balancing the extraction of global and local features, particularly in diverse and complex environments, leading to insufficient reconstruction precision and quality. We propose M3D, a novel single-view 3D reconstruction framework, to tackle these challenges. This framework adopts a dual-stream feature extraction strategy based on Selective State Spaces to effectively balance the extraction of global and local features, thereby improving scene comprehension and representation precision. Additionally, a parallel branch extracts depth information, effectively integrating visual and geometric features to enhance reconstruction quality and preserve intricate details. Experimental results indicate that the fusion of multi-scale features with depth information via the dual-branch feature extraction significantly boosts geometric consistency and fidelity, achieving state-of-the-art reconstruction performance.

M3D: Dual-Stream Selective State Spaces and Depth-Driven Framework for High-Fidelity Single-View 3D Reconstruction

TL;DR

Experimental results indicate that the fusion of multi-scale features with depth information via the dual-branch feature extraction significantly boosts geometric consistency and fidelity, achieving state-of-the-art reconstruction performance.

Abstract

The precise reconstruction of 3D objects from a single RGB image in complex scenes presents a critical challenge in virtual reality, autonomous driving, and robotics. Existing neural implicit 3D representation methods face significant difficulties in balancing the extraction of global and local features, particularly in diverse and complex environments, leading to insufficient reconstruction precision and quality. We propose M3D, a novel single-view 3D reconstruction framework, to tackle these challenges. This framework adopts a dual-stream feature extraction strategy based on Selective State Spaces to effectively balance the extraction of global and local features, thereby improving scene comprehension and representation precision. Additionally, a parallel branch extracts depth information, effectively integrating visual and geometric features to enhance reconstruction quality and preserve intricate details. Experimental results indicate that the fusion of multi-scale features with depth information via the dual-branch feature extraction significantly boosts geometric consistency and fidelity, achieving state-of-the-art reconstruction performance.

Paper Structure

This paper contains 14 sections, 13 equations, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. M3D Framework Overview
  5. Dual-Stream Feature Extraction Module
  6. Implicit Surface Modeling and Decoding
  7. Training Supervision
  8. Experiment
  9. Experimental Setup
  10. Experiment Results
  11. Ablation
  12. Ablation Experimental Setup
  13. Ablation Experimental Results
  14. Conclusion

Figures (4)

  • Figure 1: Overview of the M3D framework, showing the dual-stream architecture, selective attention module, implicit representation, and rendering network. This diagram highlights the core modules and the interaction between RGB and depth streams within the system.
  • Figure 2: Comparison of 3D reconstruction results from different methods. Columns from left to right represent the input images, results from MGNMGN, LDIFldif, InstPIFuInstPIFU, SSRSSR, our method, and the ground truth (GT). This figure demonstrates the improvement in geometric fidelity and completeness achieved by our method compared to the baseline approaches.
  • Figure 3: Texture comparison between M3D and SSR SSR, highlighting M3D's superior texture smoothness and color accuracy.
  • Figure 4: Ablation experiments comparing the performance of different configurations. F-Score values are divided by 100 for visualization purposes to align with the NC axis.