NOVUM: Neural Object Volumes for Robust Object Classification

TL;DR

NOVUM integrates explicit 3D compositional object representations into neural networks by assigning a neural object volume, composed of $K$ 3D Gaussians, to each object class and learning discriminative, class-specific Gaussian features via a contrastive objective. Inference reduces to fast Gaussian feature matching in the image feature map, while pose estimation is achieved through inverse rendering of the volume. The approach yields exceptional robustness to out-of-distribution shifts (occlusions, corruptions, and real-world nuisances) with competitive in-distribution accuracy and real-time inference, and provides interpretable visualizations via Gaussian correspondences. Overall, NOVUM demonstrates that a 3D, compositional representation can substantially improve robustness and interpretability for visual recognition while maintaining practical efficiency and enabling concurrent 3D pose estimation.

Abstract

Discriminative models for object classification typically learn image-based representations that do not capture the compositional and 3D nature of objects. In this work, we show that explicitly integrating 3D compositional object representations into deep networks for image classification leads to a largely enhanced generalization in out-of-distribution scenarios. In particular, we introduce a novel architecture, referred to as NOVUM, that consists of a feature extractor and a neural object volume for every target object class. Each neural object volume is a composition of 3D Gaussians that emit feature vectors. This compositional object representation allows for a highly robust and fast estimation of the object class by independently matching the features of the 3D Gaussians of each category to features extracted from an input image. Additionally, the object pose can be estimated via inverse rendering of the corresponding neural object volume. To enable the classification of objects, the neural features at each 3D Gaussian are trained discriminatively to be distinct from (i) the features of 3D Gaussians in other categories, (ii) features of other 3D Gaussians of the same object, and (iii) the background features. Our experiments show that NOVUM offers intriguing advantages over standard architectures due to the 3D compositional structure of the object representation, namely: (1) An exceptional robustness across a spectrum of real-world and synthetic out-of-distribution shifts and (2) an enhanced human interpretability compared to standard models, all while maintaining real-time inference and a competitive accuracy on in-distribution data.

Figures (6)

• Figure 1: Schematic overview of how NOVUM is trained. The model consists of a shared backbone (yellow) and one neural object volume for each object class (grey), which are represented as 3D Gaussians on a cuboid shape. During training, the backbone first computes feature maps of the training images. Given the class label and the 3D object pose, the backbone is trained in a contrastive manner using four types of losses: (I) To make features of the same Gaussian similar across instances (green), while at the same time making the features distinct (red) from (II) features of Gaussians from the same object, (III) background features, and (IV) features of Gaussians from other objects.
• Figure 2: Overview of the classification inference pipeline. NOVUM is composed of a backbone $\Phi$ and a set of neural object volumes represented as 3D Gaussians on a cuboid shape (green box) with colored associated features. During inference, an image is first processed by the backbone into a feature map $F$. The object class is predicted by independently matching the Gaussian features to the feature map (blue box). We color-code the detected Gaussians to highlight the interpretability of our method. Brightness shows the prediction confidence. Note that the model is only confident with the correct class even though the bus is an out-of-distribution sample. The 3D object pose can also be inferred via inverse rendering of the neural object volume (red box).
• Figure 3: (a-b) t-SNE plots comparing (a) the learned features $\mathcal{C}$ of our approach and (b) the learned vertex features $\Theta$ of NeMo. As can be seen, our contrastive loss allows a much clearer distribution of the space while keeping Gaussian features from different classes far from each other (while the low-quality clustering observed in (b) may likely originates from the ImageNet pretraining). (c-d) t-SNE plots of the mean extracted feature for each car image of the test dataset. We observe a very clear organization of the samples according to the azimuth angle for (c) our approach while this organization is completely absent in (d) other feed-forward baselines (e.g. Resnet50).
• Figure 4: Four qualitative results that were misclassified by ViT-b-16. We show for each: (left) the input image and (right) the extracted feature map and the predicted 3D pose overlaid. We color coded the features by encoding the color as a function of $\mu_k$ of the matched Gaussian $C_k$ (as done in NOCS NOCS). Hence, a smooth color gradient shows a high quality matching. In the extracted features, the brightness illustrates the confidence of matching with the Gaussian features.
• Figure S1: Plot of the concentration $\kappa$ estimates for each vertex using the training dataset (range shown is 0.5 (blue) - 1 (red)).