RWKV-PCSSC: Exploring RWKV Model for Point Cloud Semantic Scene Completion

TL;DR

RWKV-PCSSC introduces a lightweight, RWKV-based approach for point cloud semantic scene completion that seeds a coarse representation with RWKV-SG and progressively refines it through multiple RWKV-PD stages. By integrating PRWKV-based global modeling with local attention in RWKV-ATTN and a shift-enabled Spatial-Mix in PRWKV, the method achieves state-of-the-art SSC performance across indoor and outdoor datasets while drastically reducing parameters and memory compared to prior methods. The work also contributes two new datasets, NYUCAD-PC-V2 and 3D-FRONT-PC, and demonstrates strong ablations showing the necessity of both the seed generator and multi-stage refinement. Overall, RWKV-PCSSC offers a scalable, efficient solution for high-quality 3D scene completion and semantic labeling with practical implications for robotics and autonomous systems.

Abstract

Semantic Scene Completion (SSC) aims to generate a complete semantic scene from an incomplete input. Existing approaches often employ dense network architectures with a high parameter count, leading to increased model complexity and resource demands. To address these limitations, we propose RWKV-PCSSC, a lightweight point cloud semantic scene completion network inspired by the Receptance Weighted Key Value (RWKV) mechanism. Specifically, we introduce a RWKV Seed Generator (RWKV-SG) module that can aggregate features from a partial point cloud to produce a coarse point cloud with coarse features. Subsequently, the point-wise feature of the point cloud is progressively restored through multiple stages of the RWKV Point Deconvolution (RWKV-PD) modules. By leveraging a compact and efficient design, our method achieves a lightweight model representation. Experimental results demonstrate that RWKV-PCSSC reduces the parameter count by 4.18$\times$ and improves memory efficiency by 1.37$\times$ compared to state-of-the-art methods PointSSC. Furthermore, our network achieves state-of-the-art performance on established indoor (SSC-PC, NYUCAD-PC) and outdoor (PointSSC) scene dataset, as well as on our proposed datasets (NYUCAD-PC-V2, 3D-FRONT-PC).

Figures (8)

• Figure 1: (a) Compared to CasFusionNet Xu2023CasFusionNet and PointSSC yan2024pointssc, RWKV-PCSSC achieves state-of-the-art performance in both scene completion and segmentation across a variety of indoor and outdoor datasets. (b) On PointSSC datasets, RWKV-PCSSC significantly reduces parameters and memory usage while maintaining competitive performance. (c) We introduce 3D-FRONT-PC, a novel semantic scene completion dataset derived from 3D-FRONT fu20213d3d-frontfu20213d3d-future scene models. Compared to NYUCAD-PC, 3D-FRONT-PC presents more challenging tasks (i.e., retaining input point cloud coordinates in the world coordinate system), covers more categories (i.e., increased from 11 to 20), and scales significantly larger (i.e., expanded from 1,449 to 12,655 instances).
• Figure 2: (a) The architecture of the RWKV-PCSSC network is depicted as follows. The network comprises an RWKV Seed Generator (RWKV-SG) module and a sequence of N stages, each consisting of RWKV Point Deconvolution (RWKV-PD) modules. (b) Detailed description of the RWKV-SG module. (c) The details of the RWKV-PD module.
• Figure 3: (a) The detailed architecture of the RWKV-ATTN. (b) The detailed architecture of the PRWKV.
• Figure 4: The specific procedure of P-Shift.
• Figure 5: (a) compares point cloud semantic scene completion results on 3D-FRONT-PC, while (b) shows results on SSC-PC.