PointMapPolicy: Structured Point Cloud Processing for Multi-Modal Imitation Learning

TL;DR

PointMapPolicy introduces structured point maps as a regular grid of 3D points to fuse geometry with RGB data in diffusion-based imitation learning. Using xLSTM as the diffusion backbone and Cat-style per-view fusion, it achieves state-of-the-art results on RoboCasa and CALVIN while maintaining fast training and real-time inference. The approach highlights the complementary strengths of geometry and appearance in long-horizon manipulation and provides extensive ablations to guide future multi-modal 3D perception in robotics. Limitations include lack of pretrained point-map encoders and the need for more sophisticated cross-modal fusion; future work explores large-scale pretraining for point-map representations.

Abstract

Robotic manipulation systems benefit from complementary sensing modalities, where each provides unique environmental information. Point clouds capture detailed geometric structure, while RGB images provide rich semantic context. Current point cloud methods struggle to capture fine-grained detail, especially for complex tasks, which RGB methods lack geometric awareness, which hinders their precision and generalization. We introduce PointMapPolicy, a novel approach that conditions diffusion policies on structured grids of points without downsampling. The resulting data type makes it easier to extract shape and spatial relationships from observations, and can be transformed between reference frames. Yet due to their structure in a regular grid, we enable the use of established computer vision techniques directly to 3D data. Using xLSTM as a backbone, our model efficiently fuses the point maps with RGB data for enhanced multi-modal perception. Through extensive experiments on the RoboCasa and CALVIN benchmarks and real robot evaluations, we demonstrate that our method achieves state-of-the-art performance across diverse manipulation tasks. The overview and demos are available on our project page: https://point-map.github.io/Point-Map/

Figures (10)

• Figure 1: Different approaches for point cloud processing: (a) Downsampling-based methods use Furthest Point Sampling (FPS) to reduce dense point clouds to sparse representations, which PointNet then processes into compact tokens. Some variants employ FPS+KNN to generate structured point patches. (b) Feature-lifting approaches first extract 2D features from images, then project these features into 3D space, creating semantically rich 3D points. (c) Our point map method structures the point cloud as a 2D grid with the same dimensions as corresponding images, enabling direct application of efficient visual encoders to each modality independently.
• Figure 2: Overview of PMP. PMP integrates multiple modalities: language instructions encoded by a pretrained CLIP model, images processed by pretrained visual encoders, and point maps processed by visual encoders trained from scratch. Leveraging x-LSTM as its backbone architecture, PMP efficiently fuses these multi-modal inputs to generate denoised actions.
• Figure 3: Fusion methods. From left to right: Add, Cat, and Attn.
• Figure 3: The table shows average completed stages. The Max. indicates the total number of stages per task.
• Figure 4: Overview of Simulation and Real World Experiments used to test PointMapPolicy. From left to right: CALVIN Benchmark mees2022calvin, RoboCasa nasiriany2024robocasa, and our Real World Setup.