PvNeXt: Rethinking Network Design and Temporal Motion for Point Cloud Video Recognition

TL;DR

The paper tackles the high computational burden of 4D point cloud video perception caused by dense, frame-wise querying in existing methods. It introduces PvNeXt, a two-module architecture consisting of a Motion Imitator and a Single-Step Motion Encoder, which enables personalized one-shot queries by learning motion and synthesizing virtual frames for each frame. Across MSR-Action3D and NTU-RGBD, PvNeXt delivers state-of-the-art accuracy while dramatically reducing parameters and computational cost (e.g., up to $23\times$ faster inference and over $60\times$ fewer parameters), and decreases memory usage, validating both effectiveness and efficiency. This approach offers a practical, scalable solution for real-time 4D point cloud understanding with potential impact on robotics and augmented/virtual reality applications.

Abstract

Point cloud video perception has become an essential task for the realm of 3D vision. Current 4D representation learning techniques typically engage in iterative processing coupled with dense query operations. Although effective in capturing temporal features, this approach leads to substantial computational redundancy. In this work, we propose a framework, named as PvNeXt, for effective yet efficient point cloud video recognition, via personalized one-shot query operation. Specially, PvNeXt consists of two key modules, the Motion Imitator and the Single-Step Motion Encoder. The former module, the Motion Imitator, is designed to capture the temporal dynamics inherent in sequences of point clouds, thus generating the virtual motion corresponding to each frame. The Single-Step Motion Encoder performs a one-step query operation, associating point cloud of each frame with its corresponding virtual motion frame, thereby extracting motion cues from point cloud sequences and capturing temporal dynamics across the entire sequence. Through the integration of these two modules, {PvNeXt} enables personalized one-shot queries for each frame, effectively eliminating the need for frame-specific looping and intensive query processes. Extensive experiments on multiple benchmarks demonstrate the effectiveness of our method.

Figures (9)

• Figure 1: Comparisons about accuracy and inference speed between our algorithm and other baselines.
• Figure 2: Illustration of various approaches to spatio-temporal modeling. (a) Current methods typically capture motion through iterative looping processes combined with dense query operations. (b) Our proposed method captures motion via personalized one-step queries targeted at virtual frames.
• Figure 3: Overall architecture of the proposed one-step query PvNeXt workflow, composed of two key modules: (i) the Motion Imitator, which captures the motions between selected frames and their subsequent frames for each sampled point; and (ii) the Single-Step Motion Encoder, which utilizes the learned dynamics to generate synthetic frames and performs a one-step query from the original frame points to their corresponding synthetic frames to extract geometric features.
• Figure 4: Comparisons between PvNeXt and other baselines on MSR-Action3D (16-frame). The size of the pentagram in (a) denotes the parameters, the larger shape denotes higher parameters.
• Figure 5: Effect of Motion Imitator.