On-the-fly Point Feature Representation for Point Clouds Analysis

TL;DR

This work tackles the challenge of capturing explicit local geometry in point clouds by introducing On-the-fly Point Feature Representation (OPFR). OPFR combines Hierarchical Sampling, Local Reference Constructor, and Curve Feature Generator to produce curvature-aware, on-the-fly features that can be plugged into backbones like PointNet++ and Point Transformer. The method eliminates the need for normal estimation, runs with minimal overhead, and significantly improves accuracy on ModelNet40 and S3DIS, achieving state-of-the-art results when paired with strong backbones. The approach also demonstrates robustness to noise and broad applicability as a backbone-agnostic feature module for various 3D vision tasks.

Abstract

Point cloud analysis is challenging due to its unique characteristics of unorderness, sparsity and irregularity. Prior works attempt to capture local relationships by convolution operations or attention mechanisms, exploiting geometric information from coordinates implicitly. These methods, however, are insufficient to describe the explicit local geometry, e.g., curvature and orientation. In this paper, we propose On-the-fly Point Feature Representation (OPFR), which captures abundant geometric information explicitly through Curve Feature Generator module. This is inspired by Point Feature Histogram (PFH) from computer vision community. However, the utilization of vanilla PFH encounters great difficulties when applied to large datasets and dense point clouds, as it demands considerable time for feature generation. In contrast, we introduce the Local Reference Constructor module, which approximates the local coordinate systems based on triangle sets. Owing to this, our OPFR only requires extra 1.56ms for inference (65x faster than vanilla PFH) and 0.012M more parameters, and it can serve as a versatile plug-and-play module for various backbones, particularly MLP-based and Transformer-based backbones examined in this study. Additionally, we introduce the novel Hierarchical Sampling module aimed at enhancing the quality of triangle sets, thereby ensuring robustness of the obtained geometric features. Our proposed method improves overall accuracy (OA) on ModelNet40 from 90.7% to 94.5% (+3.8%) for classification, and OA on S3DIS Area-5 from 86.4% to 90.0% (+3.6%) for semantic segmentation, respectively, building upon PointNet++ backbone. When integrated with Point Transformer backbone, we achieve state-of-the-art results on both tasks: 94.8% OA on ModelNet40 and 91.7% OA on S3DIS Area-5.

Figures (4)

• Figure 1: Illustration of On-the-fly Point Feature Representation (OPFR) learning paradigm. The generation of geometric features consists of three key modules: Hierarchical Sampling, Local Reference Constructor (LRCon) and Curve Feature Generator (CFGen). These geometric features are further fed into shared-MLP followed by pooling operation, constituting the final OPFR.
• Figure 2: The workflow of Point Feature Histogram, which can be decomposed into two steps. Firstly, for one interested point (pink), each neighboring point (blue) pair is described via angles. Secondly, for each angle, its distribution over all point pairs is summarized using histograms. Here, $(\mathbf{e}_1,\mathbf{e}_2,\mathbf{e}_3)$ is the local coordinate system and $\mathbf{n}$ is the normal vector.
• Figure 3: Three-view drawing of OPFR values for an airplane. We visualize the OPFR value for 1-st channel. Blue indicates small OPFR value and red indicates large OPFR value.
• Figure 4: Ablation study on the quality of OPFR representations. We evaluate the feature quality in terms of overall accuracy (OA, %) and mean accuracy (mAcc, %). White means an improvement from PointNet++ backbone.