NeuGrasp: Generalizable Neural Surface Reconstruction with Background Priors for Material-Agnostic Object Grasp Detection

TL;DR

NeuGrasp addresses the challenge of grasping transparent and specular objects with unreliable depth by introducing background priors into a neural implicit surface reconstruction framework. It combines Reconstruction Transformers with a high level shape prior and a residual foreground enhancement to robustly recover geometry from RGB inputs in narrow viewing conditions and to generate 6-DoF grasps in an end-to-end trainable system. The method achieves state-of-the-art grasping performance in both simulated and real-world experiments without depth supervision, and a Reality Augmentation variant (NeuGrasp-RA) further closes the sim-to-real gap through fine-tuning on a small real-world dataset. These contributions offer a practical, real-time solution for material-agnostic grasping, with strong implications for cluttered and visually challenging manipulation tasks.

Abstract

Robotic grasping in scenes with transparent and specular objects presents great challenges for methods relying on accurate depth information. In this paper, we introduce NeuGrasp, a neural surface reconstruction method that leverages background priors for material-agnostic grasp detection. NeuGrasp integrates transformers and global prior volumes to aggregate multi-view features with spatial encoding, enabling robust surface reconstruction in narrow and sparse viewing conditions. By focusing on foreground objects through residual feature enhancement and refining spatial perception with an occupancy-prior volume, NeuGrasp excels in handling objects with transparent and specular surfaces. Extensive experiments in both simulated and real-world scenarios show that NeuGrasp outperforms state-of-the-art methods in grasping while maintaining comparable reconstruction quality. More details are available at https://neugrasp.github.io/.

Figures (5)

• Figure 1: Overview of NeuGrasp. We introduce a generalizable method that utilizes background priors within a neural implicit surface framework to achieve real-time scene reconstruction and material-agnostic grasping from observations within a narrow field of view.
• Figure 2: Framework of NeuGrasp. NeuGrasp leverages background priors for neural surface reconstruction and material-agnostic grasp detection. A Residual Feature Enhancement module is proposed to enhance the model attention on foreground objects instead of irrelevant background information. Through feature projection, we construct an occupancy-prior volume from residual features and a shape-prior volume from scene features. These volumes are then separately combined with their corresponding multi-view features using View Transformers. After fusion, a Ray Transformer further refines the spatial information. The final reconstructed geometry is represented as a signed distance function and converted into a radiance field. Finally, the grasping module maps the reconstructed geometry to 6-DoF grasp poses, enabling end-to-end training.
• Figure 3: Real-world Experiments. (a) Experimental settings. (b) Visualization of grasps, where orange represents positive labels and light blue represents negative labels.
• Figure 4: Comparison of View Features between GraspNeRF and NeuGrasp. It shows that with the residual feature enhancement, NeuGrasp improves focus on the foreground objects. The transparent and specular objects are clearly distinguished from the background.
• Figure 5: Visualization Results. NeuGrasp achieves a cleaner reconstruction and greater grasp diversity compared to GraspNeRF. NeuGrasp-RA, fine-tuned with a small-scale real-world data, further improves the performance, particularly in grasp diversity.