GSR: Learning Structured Reasoning for Embodied Manipulation

TL;DR

GSR tackles the challenge of long-horizon embodied manipulation by decoupling task reasoning from perceptual variability through explicit world-state transitions on grounded scene graphs. It introduces the Manip-Cognition-1.6M dataset and a two-stage training pipeline (SFT then RFT with GRPO) to enable forward state evolution, goal inference, and stepwise action reasoning within a unified framework. The approach yields strong zero-shot generalization and superior long-horizon task completion on RLBench, LIBERO, and GSR-Bench, with demonstrated robustness to perceptual noise and successful real-world deployment. By grounding decision-making in explicit world representations, GSR offers a scalable inductive bias for robust, transferable embodied reasoning in open-world manipulation scenarios.

Abstract

Despite rapid progress, embodied agents still struggle with long-horizon manipulation that requires maintaining spatial consistency, causal dependencies, and goal constraints. A key limitation of existing approaches is that task reasoning is implicitly embedded in high-dimensional latent representations, making it challenging to separate task structure from perceptual variability. We introduce Grounded Scene-graph Reasoning (GSR), a structured reasoning paradigm that explicitly models world-state evolution as transitions over semantically grounded scene graphs. By reasoning step-wise over object states and spatial relations, rather than directly mapping perception to actions, GSR enables explicit reasoning about action preconditions, consequences, and goal satisfaction in a physically grounded space. To support learning such reasoning, we construct Manip-Cognition-1.6M, a large-scale dataset that jointly supervises world understanding, action planning, and goal interpretation. Extensive evaluations across RLBench, LIBERO, GSR-benchmark, and real-world robotic tasks show that GSR significantly improves zero-shot generalization and long-horizon task completion over prompting-based baselines. These results highlight explicit world-state representations as a key inductive bias for scalable embodied reasoning.

Figures (14)

• Figure 1: In this study, we introduce GSR and the Manip-Cognition Dataset. GSR advances robotic manipulation through rational reasoning ability, while Manip-Cognition provides 1.6M samples spanning scene grounding, goal interpretation, and action planning across a wide range of atomic manipulation skills.
• Figure 2: Examples of the Manip-Cognition data including scene grounding data, planning data and goal interpretation data
• Figure 3: our evaluation comprises three components: (1) open-source benchmarks with 230 tasks from RLBench and LIBERO; (2) GSR task suites with 180 long-horizon tasks of varying difficulty targeting semantic object disambiguation, spatial-aware sequencing, and goal-conditioned generalization; and (3) real-world evaluations across diverse tasks.
• Figure 4: Comparison of model performance on RLBench, showing success rates with respect to task complexity.
• Figure 5: Evaluation on GSR-Bench via Task Progress (TP %), demonstrates that GSR achieves SOTA performance across all three evaluation aspects, with strong advantages on long-horizon tasks involving complex spatial constraints and goal reasoning.