Novel Class Discovery for Point Cloud Segmentation via Joint Learning of Causal Representation and Reasoning

TL;DR

This work tackles Novel Class Discovery for 3D point cloud segmentation by reframing the problem with a Structural Causal Model to separate intrinsic causal representations from confounding factors. It introduces a three-part framework: causal representation prototype learning to purify base-class features, a causal reasoning graph to model base-to-novel causality, and a graph-convolutional network to generate pseudo-labels for novel classes. Key contributions include the first causality-based treatment for 3D-NCD, the design of causal representation prototypes with adversarial deconfounding, and a dynamic causal graph with pruning and direction constraints guided by ICM and Causal Markov principles. Extensive experiments on 3D (SemanticKITTI, SemanticPOSS) and 2D (PASCAL-5i, COCO-20i) NCD demonstrate state-of-the-art performance and underscore the practical impact for open-world perception tasks.

Abstract

In this paper, we focus on Novel Class Discovery for Point Cloud Segmentation (3D-NCD), aiming to learn a model that can segment unlabeled (novel) 3D classes using only the supervision from labeled (base) 3D classes. The key to this task is to setup the exact correlations between the point representations and their base class labels, as well as the representation correlations between the points from base and novel classes. A coarse or statistical correlation learning may lead to the confusion in novel class inference. lf we impose a causal relationship as a strong correlated constraint upon the learning process, the essential point cloud representations that accurately correspond to the classes should be uncovered. To this end, we introduce a structural causal model (SCM) to re-formalize the 3D-NCD problem and propose a new method, i.e., Joint Learning of Causal Representation and Reasoning. Specifically, we first analyze hidden confounders in the base class representations and the causal relationships between the base and novel classes through SCM. We devise a causal representation prototype that eliminates confounders to capture the causal representations of base classes. A graph structure is then used to model the causal relationships between the base classes' causal representation prototypes and the novel class prototypes, enabling causal reasoning from base to novel classes. Extensive experiments and visualization results on 3D and 2D NCD semantic segmentation demonstrate the superiorities of our method.

Figures (6)

• Figure 1: (a) On the left, misled by the shortcut feature of ‘circular support’, learning with statistical correlation fails to identify the novel class ‘chair’ but assign the label ‘stool’ straightforwardly. On the right, causality-based learning captures the base classes' causal representation (leg) and uses it as a key element for causal reasoning, correctly predicting the novel class. (b) SCM of 3D NCD: $B$ is the base class, $N$ the novel class, $X$ the original point cloud, and $U$ the confounding factor.
• Figure 2: Overview of our method. We extract the base and novel class representations from the initial point cloud. In the base class point cloud data, there exist noisy confounding factors. We obtain causal representations through causal adversarial deconfounding and then generate base class prototypes. Meanwhile, the novel class representation also generates prototypes, which are combined with the base class causal representation prototypes as nodes to construct the graph. The causal adaptive adjacency matrix dynamically adjusts the graph structure, $\mathcal{L}_{\text{pruning}}$ aims to identify and remove ineffective edges, and $\mathcal{L}_{\text{direction}}$ ensures that the information propagation direction between nodes aligns with the actual causal path. After these causal reasoning optimizations, the updated graph is input into the GCN to generate labels for base classes and pseudo-labels for novel classes.
• Figure 3: Visualization comparison between our method, NOPS, and DASL on the SemanticPOSS and SemanticKITTI datasets. In the first row, NOPS and DASL, relying on statistical methods, fail to eliminate non-causal features, causing confusion between ‘Plants’ and ‘Building’. In the second row, they also confuse ‘Road’ with ‘Sidewalk’ and ‘Terrain’ with ‘Vegetation’. In contrast, our method achieves better results and generates high-quality pseudo labels.
• Figure 4: Ablation experiment visualization.The introduction of CRP significantly reduced the misclassification between ‘Plant’ and ‘Building.’ Subsequently, integrating CRG further alleviated the confusion between ‘Plant’ and ‘Car.’ Finally, incorporating GCPL effectively reduced the confusion among ‘Plant,’ ‘Car,’ and ‘Building.’ .
• Figure 5: Comparison of novel class Grad-CAM visualization on the PASCAL-5$^i$ dataset.