Subsecond 3D Mesh Generation for Robot Manipulation

TL;DR

This work presents a sub-second end-to-end system that generates high-quality, contextually grounded 3D meshes from a single RGB-D image for robotics. It combines open-vocabulary segmentation, a diffusion-based mesh generator accelerated by FlashVDM and hierarchical decoding, and robust point-cloud registration (RANSAC+ICP) to recover correct scale and pose. Key contributions include the integration of open-vocabulary segmentation with accelerated diffusion for fast geometry, a vector-set diffusion model with a shape VAE to efficiently produce accurate meshes, and a robust registration pipeline that preserves geometric fidelity in real scenes. The results demonstrate practical utility in real-time manipulation tasks, with sub-second mesh generation and competitive accuracy, enabling on-demand meshes to inform perception and planning in robotic systems.

Abstract

3D meshes are a fundamental representation widely used in computer science and engineering. In robotics, they are particularly valuable because they capture objects in a form that aligns directly with how robots interact with the physical world, enabling core capabilities such as predicting stable grasps, detecting collisions, and simulating dynamics. Although automatic 3D mesh generation methods have shown promising progress in recent years, potentially offering a path toward real-time robot perception, two critical challenges remain. First, generating high-fidelity meshes is prohibitively slow for real-time use, often requiring tens of seconds per object. Second, mesh generation by itself is insufficient. In robotics, a mesh must be contextually grounded, i.e., correctly segmented from the scene and registered with the proper scale and pose. Additionally, unless these contextual grounding steps remain efficient, they simply introduce new bottlenecks. In this work, we introduce an end-to-end system that addresses these challenges, producing a high-quality, contextually grounded 3D mesh from a single RGB-D image in under one second. Our pipeline integrates open-vocabulary object segmentation, accelerated diffusion-based mesh generation, and robust point cloud registration, each optimized for both speed and accuracy. We demonstrate its effectiveness in a real-world manipulation task, showing that it enables meshes to be used as a practical, on-demand representation for robotics perception and planning.

Figures (6)

• Figure 1: Our system for sub-second 3D mesh generation from RGB-D input. The system combines three stages: (1) Open-vocabulary segmentation using Florence-2 and SAM2 with depth enhancement via Depth Anything v2 (0.2s), (2) Accelerated mesh generation using FlashVDM-distilled Hunyuan3D 2.0 (0.5s), and (3) Object registration via RANSAC and ICP to align the mesh with observed point cloud (0.15s). The 0.85s total runtime marks a critical step toward real-time robotic applications.
• Figure 2: The Speed vs. Quality Trade-off in Single-Image 3D Generation. Existing methods force a choice between fast but lower-quality reconstruction and high-quality but slow generation. Our system is designed to achieve both, breaking a key barrier for real-time robotics.
• Figure 3: Effect of Depth Map Quality on Registration. (a) The source object. (b) A noisy point cloud from the raw depth camera feed results in a failed registration (misaligned overlap). (c) The cleaner, more coherent point cloud predicted by DAv2 enables a successful and accurate registration.
• Figure 4: Standard Vecset Diffusion Model (VDM) pipeline for 3D mesh generation. The process consists of four stages: (a) image encoding using DINOv2 to extract visual features, (b) iterative diffusion sampling requiring N denoising steps (typically 50+), (c) VAE decoding to convert latent vectors to a dense SDF volume, and (d) mesh extraction via marching cubes. The two primary bottlenecks preventing real-time performance are the iterative diffusion process (b) and volumetric decoding (c).
• Figure 5: Qualitative Comparison of Generated Meshes and Registration Results. Our method (left) achieves high geometric quality nearly identical to the slow H3D baseline (middle), while the fast SF3D baseline (right) produces significant artifacts.