Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GRASPLAT: Enabling dexterous grasping through novel view synthesis

Matteo Bortolon, Nuno Ferreira Duarte, Plinio Moreno, Fabio Poiesi, José Santos-Victor, Alessio Del Bue

TL;DR

GRASPLAT tackles dexterous grasping by training purely on RGB images while leveraging 3D spatial cues through differentiable 3D Gaussian Splatting (3DGS) to synthesize consistent hand-object views. A ViT-based predictor yields MANO hand parameters, which are refined through a photometric loss derived from a differentiable hand-object render, enabling end-to-end learning without full 3D scans. The framework introduces a MANO-compatible NVS model, a tailored 3DGS-based dataset (built on GraspXL with textured assets), and shows substantial gains over RGB-based baselines on GRASPLAT and YCB, including zero-shot generalization. This approach reduces data collection burdens for dexterous manipulation while delivering robust, scalable grasping performance with multi-fingered hands.

Abstract

Achieving dexterous robotic grasping with multi-fingered hands remains a significant challenge. While existing methods rely on complete 3D scans to predict grasp poses, these approaches face limitations due to the difficulty of acquiring high-quality 3D data in real-world scenarios. In this paper, we introduce GRASPLAT, a novel grasping framework that leverages consistent 3D information while being trained solely on RGB images. Our key insight is that by synthesizing physically plausible images of a hand grasping an object, we can regress the corresponding hand joints for a successful grasp. To achieve this, we utilize 3D Gaussian Splatting to generate high-fidelity novel views of real hand-object interactions, enabling end-to-end training with RGB data. Unlike prior methods, our approach incorporates a photometric loss that refines grasp predictions by minimizing discrepancies between rendered and real images. We conduct extensive experiments on both synthetic and real-world grasping datasets, demonstrating that GRASPLAT improves grasp success rates up to 36.9% over existing image-based methods. Project page: https://mbortolon97.github.io/grasplat/

GRASPLAT: Enabling dexterous grasping through novel view synthesis

TL;DR

GRASPLAT tackles dexterous grasping by training purely on RGB images while leveraging 3D spatial cues through differentiable 3D Gaussian Splatting (3DGS) to synthesize consistent hand-object views. A ViT-based predictor yields MANO hand parameters, which are refined through a photometric loss derived from a differentiable hand-object render, enabling end-to-end learning without full 3D scans. The framework introduces a MANO-compatible NVS model, a tailored 3DGS-based dataset (built on GraspXL with textured assets), and shows substantial gains over RGB-based baselines on GRASPLAT and YCB, including zero-shot generalization. This approach reduces data collection burdens for dexterous manipulation while delivering robust, scalable grasping performance with multi-fingered hands.

Abstract

Achieving dexterous robotic grasping with multi-fingered hands remains a significant challenge. While existing methods rely on complete 3D scans to predict grasp poses, these approaches face limitations due to the difficulty of acquiring high-quality 3D data in real-world scenarios. In this paper, we introduce GRASPLAT, a novel grasping framework that leverages consistent 3D information while being trained solely on RGB images. Our key insight is that by synthesizing physically plausible images of a hand grasping an object, we can regress the corresponding hand joints for a successful grasp. To achieve this, we utilize 3D Gaussian Splatting to generate high-fidelity novel views of real hand-object interactions, enabling end-to-end training with RGB data. Unlike prior methods, our approach incorporates a photometric loss that refines grasp predictions by minimizing discrepancies between rendered and real images. We conduct extensive experiments on both synthetic and real-world grasping datasets, demonstrating that GRASPLAT improves grasp success rates up to 36.9% over existing image-based methods. Project page: https://mbortolon97.github.io/grasplat/
Paper Structure (12 sections, 16 equations, 2 figures, 1 table)

This paper contains 12 sections, 16 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related work
  3. GRASPLAT
  4. Overview
  5. Architecture
  6. Loss functions for hand parameters prediction
  7. NVS photometric loss for grasping
  8. GRASPLAT dataset
  9. Experiments
  10. Experimental setup
  11. Experimental evaluation
  12. Conclusions

Figures (2)

  • Figure 1: GRASPLAT predicts 3D hand poses from a single RGB image using a frozen, denoised DINOv2 backbone. The extracted feature map is processed by a spatial transformer to estimate hand parameters, which are then refined by an MLP. During training, a feedback mechanism based on 3D Gaussian Splatting (3DGS) refines predictions by comparing rendered and real images. To accelerate training, 3DGS models are precomputed before training and then cached. The rendered scene includes both the object and the articulated hand, posed according to the predicted parameters.
  • Figure 2: Comparison of methods on the GRASPLAT (top) and YCB (bottom) datasets. Each row represents a different object. For each of the compared methods, as well as for GRASPLAT, we present the results in the simulated environment. Additionally, for our method, we also show qualitative image synthesis of our 3DGS model during training against the relative ground truth.