A Hybrid Generative and Discriminative PointNet on Unordered Point Sets

TL;DR

GDPNet presents a first hybrid Generative and Discriminative PointNet for unordered point clouds by casting classification and generation within a Joint Energy-based Model framework. It extends the JEM approach to point clouds using a PointNet backbone, defines an energy function via $E_{oldsymbol{\theta}}(\mathbf{X})$ and a joint density $p_{oldsymbol{\theta}}(\mathbf{X},y)$, and employs SGLD sampling for training. To close the gaps in classification accuracy and synthesis quality, GDPNet incorporates Sharpness-Aware Minimization (SAM) and uses a smooth activation function CELU, enabling a single model to perform well across all 10 ModelNet10 categories. Empirically, GDPNet achieves 92.8% accuracy on ModelNet10 and delivers competitive generation results compared to state-of-the-art generative models, while avoiding per-category training or separate classifiers for generation and classification.

Abstract

As point cloud provides a natural and flexible representation usable in myriad applications (e.g., robotics and self-driving cars), the ability to synthesize point clouds for analysis becomes crucial. Recently, Xie et al. propose a generative model for unordered point sets in the form of an energy-based model (EBM). Despite the model achieving an impressive performance for point cloud generation, one separate model needs to be trained for each category to capture the complex point set distributions. Besides, their method is unable to classify point clouds directly and requires additional fine-tuning for classification. One interesting question is: Can we train a single network for a hybrid generative and discriminative model of point clouds? A similar question has recently been answered in the affirmative for images, introducing the framework of Joint Energy-based Model (JEM), which achieves high performance in image classification and generation simultaneously. This paper proposes GDPNet, the first hybrid Generative and Discriminative PointNet that extends JEM for point cloud classification and generation. Our GDPNet retains strong discriminative power of modern PointNet classifiers, while generating point cloud samples rivaling state-of-the-art generative approaches.

Figures (4)

• Figure 1: GDPNet performs point cloud classification and generation with a single network. It can generate 10 categories of point clouds, while achieving a 92.8% classification accuracy on ModelNet10. Sample point clouds generated by GDPNet are provided above.
• Figure 2: Overview of GDPNet architecture. Following the design of JEM jem, PointNet is leveraged for unordered point set feature extraction, and the LogSumExp(·) of the logits from the softmax classifier can be re-used to define an energy function of point cloud $\bm{X}$, which leads to a hybrid generative and discriminative model with the fake samples generated from the SGLD sampling. The model is optimized to perform the classification and maximize the energy difference between fake and real samples.
• Figure 3: Sample point clouds generated by GDPNet. Each row corresponds to one category. The first column is a sample from ModelNet10 training set, and the rest of the columns are synthesized point clouds generated via SGLD.
• Figure 4: Sample point clouds generated by GPointNet xie2021generative and GDPNet. Our GDPNet generates chairs with more diverse styles, while GPointNet generates chairs with better details on the four legs.