Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exploiting Radiance Fields for Grasp Generation on Novel Synthetic Views

Abhishek Kashyap, Henrik Andreasson, Todor Stoyanov

TL;DR

This work investigates whether novel view synthesis from radiance fields can aid grasp generation by providing additional context beyond sparsely captured real views. By building Gaussian Splatting radiance fields from $M=3$ real views to render $N=16$ novel viewpoints, a pretrained grasp detector is applied to both real and novel views, with post-processing (pose-NMS and clustering/top-grasp filtering) to produce high-quality grasps. Experiments on the Graspnet-1billion dataset show that novel views yield additional force-closure grasps and improve grasp coverage, with notable gains in objects that lacked grasps from real views; the approach is positioned as a step toward leveraging radiance fields for grasp extraction, potentially extending to single-view or diffusion-based methods in the future. The results highlight the practical potential to reduce real-camera motion while enhancing grasp feasibility, though real-robot validation and improved view selection remain necessary for deployment.

Abstract

Vision based robot manipulation uses cameras to capture one or more images of a scene containing the objects to be manipulated. Taking multiple images can help if any object is occluded from one viewpoint but more visible from another viewpoint. However, the camera has to be moved to a sequence of suitable positions for capturing multiple images, which requires time and may not always be possible, due to reachability constraints. So while additional images can produce more accurate grasp poses due to the extra information available, the time-cost goes up with the number of additional views sampled. Scene representations like Gaussian Splatting are capable of rendering accurate photorealistic virtual images from user-specified novel viewpoints. In this work, we show initial results which indicate that novel view synthesis can provide additional context in generating grasp poses. Our experiments on the Graspnet-1billion dataset show that novel views contributed force-closure grasps in addition to the force-closure grasps obtained from sparsely sampled real views while also improving grasp coverage. In the future we hope this work can be extended to improve grasp extraction from radiance fields constructed with a single input image, using for example diffusion models or generalizable radiance fields.

Exploiting Radiance Fields for Grasp Generation on Novel Synthetic Views

TL;DR

This work investigates whether novel view synthesis from radiance fields can aid grasp generation by providing additional context beyond sparsely captured real views. By building Gaussian Splatting radiance fields from real views to render novel viewpoints, a pretrained grasp detector is applied to both real and novel views, with post-processing (pose-NMS and clustering/top-grasp filtering) to produce high-quality grasps. Experiments on the Graspnet-1billion dataset show that novel views yield additional force-closure grasps and improve grasp coverage, with notable gains in objects that lacked grasps from real views; the approach is positioned as a step toward leveraging radiance fields for grasp extraction, potentially extending to single-view or diffusion-based methods in the future. The results highlight the practical potential to reduce real-camera motion while enhancing grasp feasibility, though real-robot validation and improved view selection remain necessary for deployment.

Abstract

Vision based robot manipulation uses cameras to capture one or more images of a scene containing the objects to be manipulated. Taking multiple images can help if any object is occluded from one viewpoint but more visible from another viewpoint. However, the camera has to be moved to a sequence of suitable positions for capturing multiple images, which requires time and may not always be possible, due to reachability constraints. So while additional images can produce more accurate grasp poses due to the extra information available, the time-cost goes up with the number of additional views sampled. Scene representations like Gaussian Splatting are capable of rendering accurate photorealistic virtual images from user-specified novel viewpoints. In this work, we show initial results which indicate that novel view synthesis can provide additional context in generating grasp poses. Our experiments on the Graspnet-1billion dataset show that novel views contributed force-closure grasps in addition to the force-closure grasps obtained from sparsely sampled real views while also improving grasp coverage. In the future we hope this work can be extended to improve grasp extraction from radiance fields constructed with a single input image, using for example diffusion models or generalizable radiance fields.
Paper Structure (15 sections, 5 figures, 2 tables)

This paper contains 15 sections, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Implementation
  4. Scene representation using Gaussian Splatting
  5. Selecting real views
  6. Selecting novel views
  7. Grasp inference
  8. Grasp quality metric: force-closure
  9. Experiments and Results
  10. Scene reconstruction using Gaussian Splatting
  11. Grasp generation, aggregation, and post-processing
  12. Application of pose-NMS
  13. Application of clustering and top-grasp filtering
  14. Evaluation
  15. Conclusion

Figures (5)

  • Figure 1: Example scene reconstruction showing camera poses as frustums, all pointing down: real viewpoints in red, novel viewpoints in blue
  • Figure 2: Example real views of a scene for constructing a radiance field
  • Figure 3: Clustering and top-grasp filtering
  • Figure 4: Histogram of force-closure grasps contributed additionally by $N=16$ novel views to the 190 scenes in the Graspnet-1billion dataset
  • Figure 5: Histogram of number of objects contributed additionally by $N=16$ novel views to the grasp coverage of 190 scenes in the Graspnet-1billion dataset