Generalizable Geometric Prior and Recurrent Spiking Feature Learning for Humanoid Robot Manipulation

TL;DR

This work tackles the dual challenges of semantic-to-geometry grounding and data-efficient learning in humanoid manipulation. It introduces RGMP-S, a unified framework combining the Long-horizon Geometric-prior Skill Selector (LGSS) for geometry-aware task planning and the Recursive Adaptive Spiking Network (RASNet) for spatiotemporal, data-efficient action synthesis. The approach leverages implicit geometric priors from 2D RGB inputs, Rotary Position Embeddings, adaptive spiking dynamics, and Gaussian Mixture Models to achieve robust generalization across unseen environments and long-horizon tasks. Empirical results on ManiSkill2 and three real robotic platforms show substantial gains in generalization and data efficiency, with real-time inference and strong zero-shot transfer, highlighting practical impact for open-world robotic manipulation.

Abstract

Humanoid robot manipulation is a crucial research area for executing diverse human-level tasks, involving high-level semantic reasoning and low-level action generation. However, precise scene understanding and sample-efficient learning from human demonstrations remain critical challenges, severely hindering the applicability and generalizability of existing frameworks. This paper presents a novel RGMP-S, Recurrent Geometric-prior Multimodal Policy with Spiking features, facilitating both high-level skill reasoning and data-efficient motion synthesis. To ground high-level reasoning in physical reality, we leverage lightweight 2D geometric inductive biases to enable precise 3D scene understanding within the vision-language model. Specifically, we construct a Long-horizon Geometric Prior Skill Selector that effectively aligns the semantic instructions with spatial constraints, ultimately achieving robust generalization in unseen environments. For the data efficiency issue in robotic action generation, we introduce a Recursive Adaptive Spiking Network. We parameterize robot-object interactions via recursive spiking for spatiotemporal consistency, fully distilling long-horizon dynamic features while mitigating the overfitting issue in sparse demonstration scenarios. Extensive experiments are conducted across the Maniskill simulation benchmark and three heterogeneous real-world robotic systems, encompassing a custom-developed humanoid, a desktop manipulator, and a commercial robotic platform. Empirical results substantiate the superiority of our method over state-of-the-art baselines and validate the efficacy of the proposed modules in diverse generalization scenarios. To facilitate reproducibility, the source code and video demonstrations are publicly available at https://github.com/xtli12/RGMP-S.git.

Figures (12)

• Figure 1: Overview of our framework. By integrating geometric commonsense with spatiotemporal features, our RGMP-S encodes the robot-target spatial relationships for manipulation tasks. RGMP-S achieves a 19% performance improvement and exhibits $5\times$ greater data efficiency compared to the Diffusion Policy (DP) baseline.
• Figure 2: Pipeline of RGMP-S. Upon receiving a speech command, the robot utilizes LGSS (see \ref{['subsec:gss']} for details) to identify and localize the target object. By integrating object coordinates, shape cues (extracted via Yolov8n-seg yaseen2024yolov9 model $\phi$), and geometric-prior knowledge, the robot selects an appropriate skill from the library, where each primitive is associated with a pretrained RASNet model (see \ref{['subsec:wqkv']} for details). The selected RASNet model subsequently executes the task precisely through adaptive recursive feature extraction and GMM-based refinement.
• Figure 3: Structure of Spatial Mixing Block and Channel Mixing Block. The Spatial Mixing Block uses ADM to generate Dynamic Decay $\mathcal{W}$ recursively, and employs RoPE to introduce directional awareness relative to spatial positions. These mechanisms collectively enhance the ability of the block to integrate spatial information effectively. The Channel Mixing Block reallocates channel-wise feature responses by integrating correlations between channels. See \ref{['subsec:wqkv']} for details.
• Figure 4: Hardware configuration of our humanoid robot. The platform integrates several core subsystems: an NVIDIA Jetson AGX Orin module for onboard computation; a head-mounted RGB camera for egocentric vision; an omnidirectional microphone array for audio input; a 6-degree-of-freedom robotic arm paired with a dexterous hand for manipulation; and a multi-axis joint module for torso mobility. These components collectively enable environmental perception, physical interaction, and autonomous movement. See \ref{['sec:ExperSetup']} for details.
• Figure 5: Experimental validation within the interactive bar service scenario. Evaluation of the RGMP-S framework regarding the delivery of beverages and tissues. The training phase involves 40 expert trajectories per object category. Quantitative results demonstrate that RGMP-S attains superior success rates and robustness relative to the Diffusion Policy (DP) baseline. See \ref{['sec:sota']} for details.