Towards Space-Based Environmentally-Adaptive Grasping

TL;DR

We address the problem of reliable robotic grasping in space-like, highly variable environments where rapid learning is essential. The authors develop environmentally-adapted grammarization that fuses a one-shot exteroceptive snapshot with an episode-level context vector $e \in [-1,1]^{d_e}$ into a compact latent code $z_C=[z_q \Vert z_s \Vert e]$, and learn a context-conditioned policy with Soft Actor-Critic on GPU-accelerated simulation. Empirically, the latent+environment variant achieves fast convergence and sustained high success (e.g., sustained $S\ge 0.95$ after about $8.5\times 10^{6}$ steps) and outperforms a one-shot visual baseline, aided by mutual-information regularization that decouples the quaternion channel. The work demonstrates zero-shot adaptation to regime changes and provides a path toward robust, generalizable space grasping under extreme environmental variation.

Abstract

Robotic manipulation in unstructured environments requires reliable execution under diverse conditions, yet many state-of-the-art systems still struggle with high-dimensional action spaces, sparse rewards, and slow generalization beyond carefully curated training scenarios. We study these limitations through the example of grasping in space environments. We learn control policies directly in a learned latent manifold that fuses (grammarizes) multiple modalities into a structured representation for policy decision-making. Building on GPU-accelerated physics simulation, we instantiate a set of single-shot manipulation tasks and achieve over 95% task success with Soft Actor-Critic (SAC)-based reinforcement learning in less than 1M environment steps, under continuously varying grasping conditions from step 1. This empirically shows faster convergence than representative state-of-the-art visual baselines under the same open-loop single-shot conditions. Our analysis indicates that explicitly reasoning in latent space yields more sample-efficient learning and improved robustness to novel object and gripper geometries, environmental clutter, and sensor configurations compared to standard baselines. We identify remaining limitations and outline directions toward fully adaptive and generalizable grasping in the extreme conditions of space.

Figures (9)

• Figure 1: System overview and architecture decisions.
• Figure 2: Environmental parameter decomposition (general illustration). In the ManiSkill instantiation, the environment descriptor $e$ corresponds to the normalized episode-wise physics-parameter vector (Table \ref{['tab:env_ranges']}).
• Figure 3: Target encoding via Autoenconder architecture. The latent manifold at the "bottleneck" of the trained Neural Network is then utilized as input to the RL agent for the grammarization scenarios.
• Figure 4: Target encoding via Autoenconder architecture. The latent manifold at the "bottleneck" of the trained Neural Network is then utilized as input to the RL agent for the grammarization scenarios.
• Figure 5: Illustrative simulation scene with morphed cubic target and exteroceptive modalities used to form the one-shot observation (e.g., RGB, depth, segmentation). In our protocol, the policy receives a single snapshot at episode start and then executes open-loop with respect to exteroception.

Theorems & Definitions (5)

• Lemma 1: Local Lipschitzness of unitisation
• Proposition 1: Orientation error propagation through unitisation
• Proposition 2: Value-function bias from representation mismatch
• Corollary 1: Conditional total-variation coupling (Pinsker)
• Proposition 3: Mean-square bounded orientation error under a local small-gain condition