Common3D: Self-Supervised Learning of 3D Morphable Models for Common Objects in Neural Feature Space

TL;DR

Common3D tackles the challenge of learning 3D priors for common objects without labeled 3D data by jointly training a deformable 3D morphable model and a correspondence-aware neural feature field from object-centric videos. It uses a hybrid volumetric-mesh shape representation via φ_sdf and DMTet, with per-instance deformations φ_a conditioned on an image-derived latent code, and a neural feature appearance attached to mesh vertices, enhanced by an adapter on top of DINOv2 to improve 2D-3D correspondences. The model is trained with self-supervised geometric and appearance losses, enabling inverse rendering for pose estimation and segmentation, and a contrastive objective to avoid feature collapse. At inference, Common3D predicts the latent shape code and optimizes the 3D camera pose to maximize surface-consistent feature rendering, achieving state-of-the-art zero-shot performance on pose, segmentation, and semantic correspondence across multiple in-the-wild benchmarks. Overall, the approach unifies self-supervised representation learning with deformable 3D priors to enable robust 3D reasoning for everyday objects.

Abstract

3D morphable models (3DMMs) are a powerful tool to represent the possible shapes and appearances of an object category. Given a single test image, 3DMMs can be used to solve various tasks, such as predicting the 3D shape, pose, semantic correspondence, and instance segmentation of an object. Unfortunately, 3DMMs are only available for very few object categories that are of particular interest, like faces or human bodies, as they require a demanding 3D data acquisition and category-specific training process. In contrast, we introduce a new method, Common3D, that learns 3DMMs of common objects in a fully self-supervised manner from a collection of object-centric videos. For this purpose, our model represents objects as a learned 3D template mesh and a deformation field that is parameterized as an image-conditioned neural network. Different from prior works, Common3D represents the object appearance with neural features instead of RGB colors, which enables the learning of more generalizable representations through an abstraction from pixel intensities. Importantly, we train the appearance features using a contrastive objective by exploiting the correspondences defined through the deformable template mesh. This leads to higher quality correspondence features compared to related works and a significantly improved model performance at estimating 3D object pose and semantic correspondence. Common3D is the first completely self-supervised method that can solve various vision tasks in a zero-shot manner.

Figures (6)

• Figure 1: Common3D learns category-specific 3D morphable models from few casually captured videos completely self-supervised, and can estimate the 3D object shape (visualized from two viewpoints), 2D-3D correspondences, and the 3D object pose via inverse rendering.
• Figure 2: Method Overview. At the core of our method is a category-level template with semantic features that is acquired using a neural SDF with Differentiable Marching Tetrahedra (DMTet) where features are attached using a feature field. The category-level template is morphed using an MLP that is conditioned on a latent code $\mathrm{l}$ and rotated using the pose $\pi$. The pose $\pi$ is estimated unsupervised at training time and predicted at inference time. DINOv2 serves as a feature encoder and its output is processed in two branches: One for estimating the latent code $\mathrm{l}$ with a block of convolutional layers; and another branch serves as an adapter to enhance the DINO features for better correspondence learning. As a result, the features from the adapter are better suited for finding correspondences. For example, the tire features of DINO have a similar encoding and are, hence, ambiguous, whereas it is much more distinct after the adapter. We also visualize the training objectives: The appearance objective compares two surface probabilities: 1) The geometric probability $\mathrm{p}(\mathrm{v}_j | \mathrm{v}_i, \mathrm{V})$, as defined via the Euclidean distances of the mesh vertices; and 2) the appearance probability $\bar{\mathrm{p}}(\mathrm{f}_j | s_i, \mathrm{F}, \beta)$, acquired by comparing the 2D image features $s_i$ with the surface features $\mathrm{F}$. The remaining geometric objectives enforce the 3D shape and the projected mask to be consistent with the data acquired from a SFM pipeline.
• Figure 3: Qualitative results on the ObjectNet3D dataset. In the second row the results of our method are illustrated, in the third the results of UOP3D sommer2024unsupervised. Notably, our method fits the object more accurately, resulting in improved 3D pose and segmentation accuracy.
• Figure 4: Comprehensive qualitative results on the ObjectNet3D dataset. In the second row the results of our method are illustrated, in the third the results of UOP3D sommer2024unsupervised. Notably, our method fits the object more accurately, resulting in improved 3D pose and segmentation accuracy.
• Figure 5: Comprehensive qualitative results on the SPair-71k dataset. In the first row the results of DINOv2 are illustrated, in the third the results of our method. Our method can improve DINOv2 correspondences by resolving ambiguities in parts and symmetries.