OmniManip: Towards General Robotic Manipulation via Object-Centric Interaction Primitives as Spatial Constraints
Mingjie Pan, Jiyao Zhang, Tianshu Wu, Yinghao Zhao, Wenlong Gao, Hao Dong
TL;DR
OmniManip tackles the gap between VLM-based commonsense and precise 3D manipulation by introducing an object-centric canonical space and interaction primitives $\mathcal{O}=\{\mathbf{p},\mathbf{v}\}$ to encode where and how to interact. It implements a dual closed-loop system that plans via primitive resampling, interaction rendering, and VLM validation, and executes with real-time 6D pose tracking to optimize the end-effector pose $P^{ee*}$ under spatial and collision constraints $\mathcal{L}_C$, $\mathcal{L}_{\text{collision}}$, and $\mathcal{L}_{\text{path}}$ without fine-tuning the VLM. The approach achieves strong zero-shot generalization across 12 open-vocabulary tasks and enables automatic generation of demonstration data for imitation learning, illustrating a scalable path toward data-efficient, open-world robotic manipulation. This work provides a robust framework for bridging high-level reasoning with fine-grained 3D control in unstructured environments, with potential to accelerate large-scale robotic data generation and deployment.
Abstract
The development of general robotic systems capable of manipulating in unstructured environments is a significant challenge. While Vision-Language Models(VLM) excel in high-level commonsense reasoning, they lack the fine-grained 3D spatial understanding required for precise manipulation tasks. Fine-tuning VLM on robotic datasets to create Vision-Language-Action Models(VLA) is a potential solution, but it is hindered by high data collection costs and generalization issues. To address these challenges, we propose a novel object-centric representation that bridges the gap between VLM's high-level reasoning and the low-level precision required for manipulation. Our key insight is that an object's canonical space, defined by its functional affordances, provides a structured and semantically meaningful way to describe interaction primitives, such as points and directions. These primitives act as a bridge, translating VLM's commonsense reasoning into actionable 3D spatial constraints. In this context, we introduce a dual closed-loop, open-vocabulary robotic manipulation system: one loop for high-level planning through primitive resampling, interaction rendering and VLM checking, and another for low-level execution via 6D pose tracking. This design ensures robust, real-time control without requiring VLM fine-tuning. Extensive experiments demonstrate strong zero-shot generalization across diverse robotic manipulation tasks, highlighting the potential of this approach for automating large-scale simulation data generation.