Incentivizing Multimodal Reasoning in Large Models for Direct Robot Manipulation

TL;DR

ReasonManip addresses how to harness large multimodal models for robot manipulation by reframing manipulation as next-goal reasoning in a unified language space. It introduces an axis-based rotation representation and a two-stage training pipeline combining supervised fine-tuning and GRPO to incentivize system-2 reasoning. The 7B ReasonManip demonstrates strong out-of-distribution generalization, sim-to-real transfer, and interpretability, achieving competitive performance with data-efficient training. This approach suggests a scalable path for LMM-driven robotics where high-level reasoning directly yields low-level poses.

Abstract

Recent Large Multimodal Models have demonstrated remarkable reasoning capabilities, especially in solving complex mathematical problems and realizing accurate spatial perception. Our key insight is that these emerging abilities can naturally extend to robotic manipulation by enabling LMMs to directly infer the next goal in language via reasoning, rather than relying on a separate action head. However, this paradigm meets two main challenges: i) How to make LMMs understand the spatial action space, and ii) How to fully exploit the reasoning capacity of LMMs in solving these tasks. To tackle the former challenge, we propose a novel task formulation, which inputs the current states of object parts and the gripper, and reformulates rotation by a new axis representation instead of traditional Euler angles. This representation is more compatible with spatial reasoning and easier to interpret within a unified language space. For the latter challenge, we design a pipeline to utilize cutting-edge LMMs to generate a small but high-quality reasoning dataset of multi-round dialogues that successfully solve manipulation tasks for supervised fine-tuning. Then, we perform reinforcement learning by trial-and-error interactions in simulation to further enhance the model's reasoning abilities for robotic manipulation. Our resulting reasoning model built upon a 7B backbone, named ReasonManip, demonstrates three notable advantages driven by its system-2 level reasoning capabilities: i) exceptional generalizability to out-of-distribution environments, objects, and tasks; ii) inherent sim-to-real transfer ability enabled by the unified language representation shared across domains; iii) transparent interpretability connecting high-level reasoning and low-level control. Extensive experiments demonstrate the effectiveness of the proposed paradigm and its potential to advance LMM-driven robotic manipulation.

Figures (16)

• Figure 1: An illustration of ReasonManip that solves robot manipulation tasks by next-goal prediction via unified system-2 level reasoning, including status infering, collision check, failure mode discovery, task replanning and reflection, etc. Unlike prior approaches zawalski2024robotic that rely on separate high-level task decomposition and low-level action models or action heads, ReasonManip enables LMMs to "directly" infer the next goal in natural language (e.g. target gripper poses) via multimodal reasoning.
• Figure 2: Overall pipeline of our method. On the first stage, we harness the advanced LMM-72B to generate robot manipulation reasoning data in the form of iteratively and interactive multi-round conversations with the users, which is used for SFT on a smaller LMM-7B model. On the second stage, the model interact with the virtual environment for GRPO training to further incentivize its reasoning ability for robot manipulation.
• Figure 3: Example of the axis representation for the orientation of gripper (top left) and objects (top middle), which is in most cases more straightforward and manipulable for LMMs than the Euler angle representation (bottom left two subplots).
• Figure 4: The illustration of our multi-round conversation collection pipeline. The predicted gripper pose is executed in a simulator to obtain the updated observation and scene information. We utilize human-written guidance to help model reason and generate the next-goal prediction.
• Figure 5: Illustration of the environment comparisons of the SIMPLER and the MetaWorld environment. We can see that they have large gaps in camera views, robots, object visuals, and backgrounds.