ManiTrend: Bridging Future Generation and Action Prediction with 3D Flow for Robotic Manipulation

TL;DR

3D flow is highlighted as an effective bridge between language-based future image generation and fine-grained action prediction, and ManiTrend is developed, a unified framework that models the dynamics of 3D particles, vision observations and manipulation actions with a causal transformer.

Abstract

Language-conditioned manipulation is a vital but challenging robotic task due to the high-level abstraction of language. To address this, researchers have sought improved goal representations derived from natural language. In this paper, we highlight 3D flow - representing the motion trend of 3D particles within a scene - as an effective bridge between language-based future image generation and fine-grained action prediction. To this end, we develop ManiTrend, a unified framework that models the dynamics of 3D particles, vision observations and manipulation actions with a causal transformer. Within this framework, features for 3D flow prediction serve as additional conditions for future image generation and action prediction, alleviating the complexity of pixel-wise spatiotemporal modeling and providing seamless action guidance. Furthermore, 3D flow can substitute missing or heterogeneous action labels during large-scale pretraining on cross-embodiment demonstrations. Experiments on two comprehensive benchmarks demonstrate that our method achieves state-of-the-art performance with high efficiency. Our code and model checkpoints will be available upon acceptance.

Figures (8)

• Figure 1: An overview of ManiTrend, the proposed end-to-end framework that tracks the dynamics of 3D particles, vision observations and manipulation actions in a unified manner. Better viewed in color.
• Figure 2: Environments for our experiments. CALVIN involves 34 manipulation tasks and 4 scenes of different colors, textures and object placements. LIBERO features 4 evaluation task suites that challenge different dimensions of capability.
• Figure 3: Performance comparison on the 4 task suites of LIBERO benchmark. For each task suite, the success rate is calculated over 50 rollouts for each task within the task suite. The overall success rate is averaged over the results on the four task suites.
• Figure 4: Performance on CALVIN (ABC$\rightarrow$D) of MainTrend pretrained with different amounts of data.
• Figure 5: Visualization of predicted 3D flow on CALVIN (D$\rightarrow$D). The images in the upper row are main-view observations with rendered flow (only x, y are considered). Whereas the images in lower row visualize 3D flow in 3D space (for visual clarity, we only visualize the flow of moving particles that get sampled).