Towards Training-Free Open-World Classification with 3D Generative Models

TL;DR

This work tackles 3D open-world classification with open-category and open-pose challenges by proposing a training-free framework that leverages 3D generative priors to synthesize rotation-invariant anchor samples. A rotation-invariant 3D encoder, combined with LLM-derived category descriptions and diffusion-based 3D generation, forms a lightweight, retraining-free classifier whose predictions rely on cosine similarity to anchor prototypes. The approach delivers state-of-the-art improvements on ModelNet10 and McGill, demonstrates robustness to pose variation, and analyzes how anchor quantity, generative models, and representation backbones influence performance. It also explores the role of LLM prompts in generating category descriptions, highlighting both potential gains and challenges in prompt reliability and prompt-based diversity.

Abstract

3D open-world classification is a challenging yet essential task in dynamic and unstructured real-world scenarios, requiring both open-category and open-pose recognition. To address these challenges, recent wisdom often takes sophisticated 2D pre-trained models to provide enriched and stable representations. However, these methods largely rely on how 3D objects can be projected into 2D space, which is unfortunately not well solved, and thus significantly limits their performance. Unlike these present efforts, in this paper we make a pioneering exploration of 3D generative models for 3D open-world classification. Drawing on abundant prior knowledge from 3D generative models, we additionally craft a rotation-invariant feature extractor. This innovative synergy endows our pipeline with the advantages of being training-free, open-category, and pose-invariant, thus well suited to 3D open-world classification. Extensive experiments on benchmark datasets demonstrate the potential of generative models in 3D open-world classification, achieving state-of-the-art performance on ModelNet10 and McGill with 32.0% and 8.7% overall accuracy improvement, respectively.

Figures (5)

• Figure 1: Comparison of Pipelines: (a) SOTA methods open-poseZhu2022PointCLIPV2 project 3D samples into 2D images to harness 2D prior knowledge for novel category recognition but are sensitive to pose changes, (b) The proposed pipeline applies 3D prior knowledge to generate anchor samples for novel categories, embedding them in a rotation-invariant space for effective performance in open-pose scenarios. Predictions are computed via feature similarity between test and anchor samples.
• Figure 2: Overview of the network architecture with two components. (1) Open-world classifier: Category names are input into ChatGPT to create descriptions, utilized by a pre-trained text-to-3D model to generate anchor samples. These samples are augmented to align with test samples. A pre-trained 3D model extracts features, marking them as prototypes for classification. (2) Test sample inference: Test samples are augmented and processed through the same model for feature extraction. Predictions are based on cosine similarity between test and anchor sample features. Different colors denote new categories, gray represents test data.
• Figure 3: Comparison on the Number of Anchor Samples $N_a$.
• Figure 4: t-SNE visualizations of feature representations extracted by TAP and TET under aligned-pose and open-pose settings on the ModelNet10 dataset. Different colors indicate different object categories according to the ground truth labels. Triangles represent generated anchor samples, while circles denote original dataset samples.
• Figure 5: Visualization comparison of "dresser" point clouds from original category names ($\mathbf{CN}$) and ChatGPT-generated prompts ($\mathbf{CD}$), which enhance anchors' diversity.