GPA-RAM: Grasp-Pretraining Augmented Robotic Attention Mamba for Spatial Task Learning

TL;DR

GPA-RAM presents a two-part approach to robust, real-time robotic manipulation by coupling Grasp-Pretraining Augmentation (GPA) with Robotic Attention Mamba (RAM). GPA injects pretrained grasp priors to preserve fine-grained manipulation cues, while RAM leverages linear-time state-space modeling to achieve efficient spatial perception and inference. Across RLBench, ALOHA, and real-world setups, GPA-RAM delivers state-of-the-art or competitive gains in both discrete and continuous tasks, with inference speeds around 71 FPS. This framework demonstrates that combining grasp priors with efficient spatial modeling yields precise, responsive robotic systems suitable for diverse multi-task scenarios.

Abstract

Task failures in prior fine-grained robotic manipulation methods often stem from suboptimal initial grasping, which is critical for subsequent manipulation and reducing the requirement for complex pose adjustments. To address this, we propose Grasp-Pretraining Augmentation (GPA), a general multi-modal learning framework that enhances grasp perception without additional grasp pose data collection and labeling. GPA achieves evident enhancement on RLBench multi-task benchmark (from 79.3% to 84.2%) and ALOHA bimanual manipulation tasks (from 86%/16% to 98%/38%). Although GPA enhances fine-grained grasping performance by leveraging increased model capacity, it incurs computational latency and hinders real-time deployment. To mitigate this limitation, we propose Robotic Attention Mamba (RAM). This architecture synergizes attention mechanisms with state space models (SSMs), effectively capturing complex spatial features while maintaining superior inference efficiency. Our unified GPA-RAM framework balances model capacity with efficiency and applies to both discrete and continuous action generation. GPA-RAM demonstrates superior performance across four robotic systems with diverse camera configurations in both simulation and the real world. Compared with previous state-of-the-art methods, it improves average success rates by 8.2% over RVT2 (from 79.3% to 87.5%) and 2.6% over ARP^+ (from 84.9% to 87.5%) on the RLBench multi-task benchmark and 40% (from 16% to 56%), 12% (from 86% to 98%) on ALOHA bimanual continuous tasks, with inference speed of about 71 FPS. This work provides a framework for developing robotic systems that are simultaneously precise and responsive. The project and code are at https://gpa-ram.github.io/

Figures (8)

• Figure 1: Previous imitation learning methods struggle with tasks requiring high-precision manipulation and robust spatial perception. This limitation leads to failures such as grasping errors in "Stack the other cups on top of the maroon cup" (top row), imprecise initial grasping and insertion in "Insert the square ring onto the red peg" (middle row), collision caused by poor spatial perception in "Put groceries in the cupboard" (bottom row). In contrast, the proposed Grasp-Pretraining Augmented Robotic Attention Mamba (GPA-RAM) achieves precise manipulation and superior spatial perception.
• Figure 2: Overview of the GPA-RAM on RLBench. The GPA-RAM primarily comprises two modules: RAM for spatial features extraction and GPA for grasping skills. It processes point clouds, RGB-D, language instructions, and robot proprioception, extracting spatial semantic features through Coarse and Fine Robotic Attention Mamba in RAM and grasping features via the pre-trained & frozen Grasp Pose Detector in GPA. These features are fused in GPA's Pre-trained Location Fusion to predict the next key action in 3D space.
• Figure 3: Network Architecture of Robotic Attention Mamba and Grasp-Pretraining Augmentation. Mamba (Left): Details of the Mamba Block. one-stage RAM (Middle): Visual and robot proprioceptive observations are processed using parallel single-view attention to extract features, which are merged into multi-view and language features for contrastive learning via spatial Mamba blocks. Residual layers, layer normalization, and feedforward layers are applied afterward. GPA (Right): Visual and language features are learned by the grasp pose detector, with spatial attention fusing features from RAM, followed by upsampling and MLP layers for action prediction.
• Figure 4: Adapt GPA and One-Stage RAM to Continuous Actions Generation.
• Figure 5: Qualitative Comparison of the previous SOTA (ARP$^+$) and our GPA-RAM on RLBench Tasks. In the first row, ARP$^+$ fails in the "Insert the square ring onto the orange peg" task due to an inaccurate initial grasping pose and insufficient pose adjustment. In the third row, it fails in the "Sort the cyan cube into the cube slot of the shape sorter" task because of the erroneous initial grasping pose.