Learning Spatial-Aware Manipulation Ordering

TL;DR

OrderMind presents a unified framework for spatial-aware manipulation ordering in cluttered environments, combining a spatial context encoder with a temporal priority structuring module to directly predict manipulation priorities. It constructs a kNN-based spatial graph to capture object-object and object–manipulator interactions and introduces spatial priors, generated via a Vision-Language Model, to supervise learning without manual labeling. The approach achieves state-of-the-art performance on a large Manipulation Ordering Benchmark, delivering real-time inference and robust transfer to real-world scenes, with ablations confirming the complementary benefits of perception, context integration, and order reasoning. The work advances autonomous robotic manipulation by enabling safe, efficient, and scalable ordering in visually complex cluttered environments.

Abstract

Manipulation in cluttered environments is challenging due to spatial dependencies among objects, where an improper manipulation order can cause collisions or blocked access. Existing approaches often overlook these spatial relationships, limiting their flexibility and scalability. To address these limitations, we propose OrderMind, a unified spatial-aware manipulation ordering framework that directly learns object manipulation priorities based on spatial context. Our architecture integrates a spatial context encoder with a temporal priority structuring module. We construct a spatial graph using k-Nearest Neighbors to aggregate geometric information from the local layout and encode both object-object and object-manipulator interactions to support accurate manipulation ordering in real-time. To generate physically and semantically plausible supervision signals, we introduce a spatial prior labeling method that guides a vision-language model to produce reasonable manipulation orders for distillation. We evaluate OrderMind on our Manipulation Ordering Benchmark, comprising 163,222 samples of varying difficulty. Extensive experiments in both simulation and real-world environments demonstrate that our method significantly outperforms prior approaches in effectiveness and efficiency, enabling robust manipulation in cluttered scenes.

Figures (7)

• Figure 1: Comparison of various designs for manipulation ordering. (a) Heuristic pipelines use hand-crafted optimization algorithms for post-processing, but struggle with generalization. (b) Two-stage frameworks utilize separate networks to predict manipulation order, resulting in increased latency and reduced efficiency. (c) Our unified spatial-aware framework jointly learns spatial representation and manipulation ordering, achieving both high accuracy and real-time performance.
• Figure 2: Main framework of OrderMind. OrderMind captures spatially-aware manipulation ordering representations that encode both object-object relationships and object-manipulator interactions in cluttered environments. The spatial context understanding module aggregates spatial information from local geometry, while the temporal priority structuring module integrates local object features with global scene context. OrderMind predicts the manipulation priority for each object, and by sorting these priorities, we can obtain the manipulation order.
• Figure 3: Details of Spatial Context Understanding Module. A spatial graph is constructed from object-centric point clouds to capture scene layout in cluttered environments. Local geometric structure is extracted using a k-Nearest Neighbors strategy, and spatial-aware features are aggregated through neighborhood pooling to support manipulation ordering.
• Figure 4: Visualization of manipulation ordering results under easy, moderate, and hard settings. 3D bounding boxes are demonstrated and manipulation orders are marked at the center of each object.
• Figure 5: Visualization of real-world manipulation ordering. The left figure shows results from experiments conducted in a laboratory setting, while the right figure presents results obtained in an actual factory environment. Predicted order is annotated on each object in the scene.