RVT: Robotic View Transformer for 3D Object Manipulation

TL;DR

RVT tackles the scalability bottleneck of explicit 3D representations in robotic manipulation by using a multi-view transformer over re-rendered virtual views. It decouples input rendering from the transformer, enabling efficient multi-view 3D reasoning and effective use of language goals via CLIP embeddings. In RLBench with 18 tasks, RVT outperforms state-of-the-art voxel-based methods in success rate while cutting training time by orders of magnitude and doubling inference speed, and it demonstrates real-world manipulation with a small demonstration budget. The work suggests that re-rendered, orthographic multi-view inputs with structured attention offers a practical path toward scalable, accurate 3D manipulation.

Abstract

For 3D object manipulation, methods that build an explicit 3D representation perform better than those relying only on camera images. But using explicit 3D representations like voxels comes at large computing cost, adversely affecting scalability. In this work, we propose RVT, a multi-view transformer for 3D manipulation that is both scalable and accurate. Some key features of RVT are an attention mechanism to aggregate information across views and re-rendering of the camera input from virtual views around the robot workspace. In simulations, we find that a single RVT model works well across 18 RLBench tasks with 249 task variations, achieving 26% higher relative success than the existing state-of-the-art method (PerAct). It also trains 36X faster than PerAct for achieving the same performance and achieves 2.3X the inference speed of PerAct. Further, RVT can perform a variety of manipulation tasks in the real world with just a few ($\sim$10) demonstrations per task. Visual results, code, and trained model are provided at https://robotic-view-transformer.github.io/.

Figures (5)

• Figure 1: RVT scales and performs better than PerAct on RLBench, achieving on-par performance in 36X less time (same hardware), and 1.26X peak performance.
• Figure 2: Overview of RVT. Given RGB-D from sensor(s), we first construct a point cloud of the scene. The point cloud is then used to produce virtual images around the robot workspace. The virtual images are fed to a multi-view transformer model to predict view-specific features, which are then combined to predict action in 3D.
• Figure 3: We evaluate RVT with various camera locations for re-rendering (a-d) and find that locations in (a) perform best. We also test various projection options (e-f) for rendering images and find that RVT works better with orthographic images.
• Figure 4: Examples of RVT in the real world. A single RVT model can perform multiple tasks (5 tasks, 13 variations) in the real world with just $\sim$10 demonstrations per task.
• Figure 5: Overview of the transformer used in RVT. The input to the transformer is a language description of the task and virtual images of the scene point cloud. The text is converted into token embeddings using the pretrained CLIP radford2021learning model, while the virtual images are converted into token embeddings via patchify and projection operations. For each virtual image, tokens belonging to the same image are processed via four attention layers. Finally, the processed image tokens as well as the language tokens are jointly processed using four attention layers. The 3D action is inferred using the resulting image tokens.