Surfer: Progressive Reasoning with World Models for Robotic Manipulation

TL;DR

A novel and simple robot manipulation framework that treats robot manipulation as a state transfer of the visual scene, and decouples it into two parts: action and scene, and can provide a standardized testing platform for embedded AI agents in multi-modal environments.

Abstract

Considering how to make the model accurately understand and follow natural language instructions and perform actions consistent with world knowledge is a key challenge in robot manipulation. This mainly includes human fuzzy instruction reasoning and the following of physical knowledge. Therefore, the embodied intelligence agent must have the ability to model world knowledge from training data. However, most existing vision and language robot manipulation methods mainly operate in less realistic simulator and language settings and lack explicit modeling of world knowledge. To bridge this gap, we introduce a novel and simple robot manipulation framework, called Surfer. It is based on the world model, treats robot manipulation as a state transfer of the visual scene, and decouples it into two parts: action and scene. Then, the generalization ability of the model on new instructions and new scenes is enhanced by explicit modeling of the action and scene prediction in multi-modal information. In addition to the framework, we also built a robot manipulation simulator that supports full physics execution based on the MuJoCo physics engine. It can automatically generate demonstration training data and test data, effectively reducing labor costs. To conduct a comprehensive and systematic evaluation of the robot manipulation model in terms of language understanding and physical execution, we also created a robotic manipulation benchmark with progressive reasoning tasks, called SeaWave. It contains 4 levels of progressive reasoning tasks and can provide a standardized testing platform for embedded AI agents in multi-modal environments. On average, Surfer achieved a success rate of 54.74% on the defined four levels of manipulation tasks, exceeding the best baseline performance of 47.64%.

Figures (5)

• Figure 1: (a) A brief comparison of RT-1 and Surfer. (b) Comparison of manipulation success rates of different models on four progressive reasoning tasks.
• Figure 2: Overall framework of Surfer. It mainly contains two modules: action prediction and scene prediction.
• Figure 3: In the SeaWave benchmark, the proposed general pipeline mainly consists of three parts: automatic scene generation, instruction generation, and robotic manipulation.
• Figure 4: Overview of SeaWave benchmark. (a) The green box indicates the target object that the current instruction requires the robot to grasp. There is only a single object in the level 1 scene, and multiple objects in the level 2, 3, and 4 scenes. The difficulty of the four levels of tasks increases in sequence. In particular, level 4 requires a deep integration of vision and language information to make accurate decisions. (b) SeaWave’s object library contains the most common objects and supports a variety of robot manipulation scenarios.
• Figure 5: (a) Manipulation instances of RT-1 and Surfer on level 4 tasks. They evaluated the model's manipulation and reasoning abilities in terms of position, space, and appearance. Among them, the object with a background box is the target object of the current instruction. (b) The ablation experiments of scene prediction, module merging, and feature concatenation.