3D Congealing: 3D-Aware Image Alignment in the Wild

TL;DR

3D Congealing introduces the task of aligning unlabeled, semantically similar objects across diverse internet images into a shared canonical $3D$ representation, yielding dense 2D–3D correspondences and per-image poses. The method fuses a pre-trained text-to-image diffusion prior via Textual Inversion and Score Distillation Sampling with semantic feature consistency (e.g., $d_ abla$ from $f_ eta$) to steer optimization toward a plausible canonical shape while remaining robust to appearance and illumination variations. It defines forward and reverse canonical coordinate mappings to establish dense 2D–2D and 2D–3D correspondences, enabling pose estimation and image editing in the wild. Empirical results on multi-illumination datasets and cross-domain internet imagery show strong pose estimation performance, robust editing capabilities, and notable resilience to domain shifts, highlighting the practical impact of grounding in a canonical 3D space for unconstrained image collections.

Abstract

We propose 3D Congealing, a novel problem of 3D-aware alignment for 2D images capturing semantically similar objects. Given a collection of unlabeled Internet images, our goal is to associate the shared semantic parts from the inputs and aggregate the knowledge from 2D images to a shared 3D canonical space. We introduce a general framework that tackles the task without assuming shape templates, poses, or any camera parameters. At its core is a canonical 3D representation that encapsulates geometric and semantic information. The framework optimizes for the canonical representation together with the pose for each input image, and a per-image coordinate map that warps 2D pixel coordinates to the 3D canonical frame to account for the shape matching. The optimization procedure fuses prior knowledge from a pre-trained image generative model and semantic information from input images. The former provides strong knowledge guidance for this under-constraint task, while the latter provides the necessary information to mitigate the training data bias from the pre-trained model. Our framework can be used for various tasks such as correspondence matching, pose estimation, and image editing, achieving strong results on real-world image datasets under challenging illumination conditions and on in-the-wild online image collections.

Figures (13)

• Figure 1: Objects with different shapes and appearances, such as these sculptures, may share similar semantic parts and a similar geometric structure. We study 3D Congealing, inferring and aligning such a shared structure from an unlabeled image collection. Such alignment can be used for tasks such as pose estimation and image editing. See \ref{['app:rodin_full']} for full results.
• Figure 1: Overview.
• Figure 2: Pipeline. Given a collection of in-the-wild images capturing similar objects as inputs, we develop a framework that "congeals" these images in 3D. The core representation consists of a canonical 3D shape that captures the geometric structure shared among the inputs, together with a set of coordinate mappings that register the input images to the canonical shape. The framework utilizes the prior knowledge of plausible 3D shapes from a generative model, and aligns images in the semantic space using pre-trained semantic feature extractors.
• Figure 3: Pose Estimation from Multi-Illumination Captures. The figure shows 4 example scenes from the NAVI dataset, displaying the real image inputs, canonical shapes under estimated poses, and the canonical coordinate maps.
• Figure 4: Pose Estimation for Tourist Landmarks. This is a challenging problem setting due to the varying viewpoints and lighting conditions, and the proposed method can successfully align online tourist photos taken at different times and possibly at different geographical locations, into one canonical representation.