Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-Modal 3D Mesh Reconstruction from Images and Text

Melvin Reka, Tessa Pulli, Markus Vincze

TL;DR

This work tackles reconstructing accurate 3D meshes of unseen objects from a few images without relying on preexisting CAD models or extensive training. It introduces a language guided few-shot pipeline that uses GroundingDINO and SAM to obtain object masks from a text prompt, feeds masked images into VGGSfM for sparse structure-from-motion, and applies Gaussian Splatting via SuGAR to produce a textured mesh, followed by artifact cleaning. The main contributions are the integration of language guided segmentation with a differentiable reconstruction pipeline and a systematic study of imaging conditions on reconstruction quality and efficiency. The approach enables practical 3D reconstruction in scenarios where obtaining CAD models is impractical and demonstrates favorable tradeoffs between accuracy, texture fidelity, and computational cost.

Abstract

6D object pose estimation for unseen objects is essential in robotics but traditionally relies on trained models that require large datasets, high computational costs, and struggle to generalize. Zero-shot approaches eliminate the need for training but depend on pre-existing 3D object models, which are often impractical to obtain. To address this, we propose a language-guided few-shot 3D reconstruction method, reconstructing a 3D mesh from few input images. In the proposed pipeline, receives a set of input images and a language query. A combination of GroundingDINO and Segment Anything Model outputs segmented masks from which a sparse point cloud is reconstructed with VGGSfM. Subsequently, the mesh is reconstructed with the Gaussian Splatting method SuGAR. In a final cleaning step, artifacts are removed, resulting in the final 3D mesh of the queried object. We evaluate the method in terms of accuracy and quality of the geometry and texture. Furthermore, we study the impact of imaging conditions such as viewing angle, number of input images, and image overlap on 3D object reconstruction quality, efficiency, and computational scalability.

Multi-Modal 3D Mesh Reconstruction from Images and Text

TL;DR

This work tackles reconstructing accurate 3D meshes of unseen objects from a few images without relying on preexisting CAD models or extensive training. It introduces a language guided few-shot pipeline that uses GroundingDINO and SAM to obtain object masks from a text prompt, feeds masked images into VGGSfM for sparse structure-from-motion, and applies Gaussian Splatting via SuGAR to produce a textured mesh, followed by artifact cleaning. The main contributions are the integration of language guided segmentation with a differentiable reconstruction pipeline and a systematic study of imaging conditions on reconstruction quality and efficiency. The approach enables practical 3D reconstruction in scenarios where obtaining CAD models is impractical and demonstrates favorable tradeoffs between accuracy, texture fidelity, and computational cost.

Abstract

6D object pose estimation for unseen objects is essential in robotics but traditionally relies on trained models that require large datasets, high computational costs, and struggle to generalize. Zero-shot approaches eliminate the need for training but depend on pre-existing 3D object models, which are often impractical to obtain. To address this, we propose a language-guided few-shot 3D reconstruction method, reconstructing a 3D mesh from few input images. In the proposed pipeline, receives a set of input images and a language query. A combination of GroundingDINO and Segment Anything Model outputs segmented masks from which a sparse point cloud is reconstructed with VGGSfM. Subsequently, the mesh is reconstructed with the Gaussian Splatting method SuGAR. In a final cleaning step, artifacts are removed, resulting in the final 3D mesh of the queried object. We evaluate the method in terms of accuracy and quality of the geometry and texture. Furthermore, we study the impact of imaging conditions such as viewing angle, number of input images, and image overlap on 3D object reconstruction quality, efficiency, and computational scalability.

Paper Structure

This paper contains 13 sections, 6 equations, 7 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. 6D Object Pose Estimation
  4. Few-Shot 3D Reconstruction Methods
  5. Language Guided Object Segmentation
  6. Method
  7. Experiments
  8. Implementation details:
  9. Evaluation Metrics
  10. Experimental Setup
  11. Dataset
  12. Evaluation
  13. SUMMARY AND OUTLOOK

Figures (7)

  • Figure 1: Images of an object, accompanied by a descriptive text prompt, are processed through the pipeline. A sparse reconstruction using COLMAP via VGGSfM is then performed to generate and refine the mesh with SuGaR.
  • Figure 2: Spherical coordinates, where we refer to the polar angle $\theta$ as viewing angle, and to the azimuthal angle $\phi$ as
  • Figure 3: THU Multi-view stereo datasets Shuji_SAKAI20152014EDP7409 of a cat and a dog. The left side shows the input images captured from various viewpoints, while the right side displays the corresponding camera viewpoints and the target objects.
  • Figure 4: Runtime vs. input images for $\theta = 45^\circ, \Delta\phi = 10^\circ$
  • Figure 5: IoU and Chamfer Distance for cat figurine at different $\theta$ angles.
  • ...and 2 more figures