PointNet4D: A Lightweight 4D Point Cloud Video Backbone for Online and Offline Perception in Robotic Applications

TL;DR

PointNet4D introduces a lightweight, unified 4D backbone for online and offline 4D point cloud video perception, combining a hybrid Mamba-Transformer temporal fusion with a frame-wise 4D masked autoregressive pretraining (4DMAP). It demonstrates strong improvements across 9 tasks on 7 datasets and delivers practical robotic gains via 4D Diffusion Policy (DP4) and 4D Imitation Learning (4DIL) on RoboTwin and HandoverSim benchmarks. The paper shows that the approach yields robust online performance while maintaining strong offline accuracy, with minimal computational overhead and broad plug-and-play compatibility with other backbones. These results highlight the viability of unified online/offline 4D perception for real-time robotics and embodied AI applications.

Abstract

Understanding dynamic 4D environments-3D space evolving over time-is critical for robotic and interactive systems. These applications demand systems that can process streaming point cloud video in real-time, often under resource constraints, while also benefiting from past and present observations when available. However, current 4D backbone networks rely heavily on spatiotemporal convolutions and Transformers, which are often computationally intensive and poorly suited to real-time applications. We propose PointNet4D, a lightweight 4D backbone optimized for both online and offline settings. At its core is a Hybrid Mamba-Transformer temporal fusion block, which integrates the efficient state-space modeling of Mamba and the bidirectional modeling power of Transformers. This enables PointNet4D to handle variable-length online sequences efficiently across different deployment scenarios. To enhance temporal understanding, we introduce 4DMAP, a frame-wise masked auto-regressive pretraining strategy that captures motion cues across frames. Our extensive evaluations across 9 tasks on 7 datasets, demonstrating consistent improvements across diverse domains. We further demonstrate PointNet4D's utility by building two robotic application systems: 4D Diffusion Policy and 4D Imitation Learning, achieving substantial gains on the RoboTwin and HandoverSim benchmarks.

Figures (4)

• Figure 1: (a) We propose PointNet4D, a lightweight and unified network that can simultaneously handle both online real-time and offline perception tasks. (b) We introduce the 4DMAP pre-training method for PointNet4D to fully harness its potential. (c) We propose the 4D Diffusion Policy and 4D Imitation Learning based on PointNet4D to enhance the robot's perception capabilities.
• Figure 2: Method Overview. The left side of the diagram details the MTMT hybrid model's internal structure, a composite of PointNet++, Mamba, Transformer, and MLP designed for processing point cloud video tasks under both online and offline conditions. The right side outlines the 4DMAP pre-training strategy, which utilizes a frame-masking technique tailored for PointNet4D to maximize its performance potential across a broad spectrum of applications. This figure is adapted from shen2023maskedwen2022pointwang2024genh2rmu2024robotwinzhang2023complete.
• Figure 3: The comparison between the 3D Diffusion Policy (DP3) ze20243d and the 4D Diffusion Policy (DP4).
• Figure 4: Comparison of our 4DIL with GenH2R wang2024genh2r.