pix2gestalt: Amodal Segmentation by Synthesizing Wholes

TL;DR

Pix2gestalt tackles zero-shot amodal segmentation by learning to synthesize the full appearance of occluded objects using a conditional diffusion model fine-tuned on a large synthetic paired dataset. By conditioning on both the observed image and a prompt, the model $f_\theta$ generates plausible wholes, enabling downstream tasks such as segmentation, recognition, and 3D reconstruction. The approach yields state-of-the-art performance on amodal benchmarks without COCO-A training, demonstrates robust generalization to art and real-world scenes, and provides a practical drop-in module to improve occlusion handling in existing vision systems. This synthesis-first, open-world strategy highlights the value of diffusion priors and synthetic data for resolving inherently ambiguous occlusions in diverse settings.

Abstract

We introduce pix2gestalt, a framework for zero-shot amodal segmentation, which learns to estimate the shape and appearance of whole objects that are only partially visible behind occlusions. By capitalizing on large-scale diffusion models and transferring their representations to this task, we learn a conditional diffusion model for reconstructing whole objects in challenging zero-shot cases, including examples that break natural and physical priors, such as art. As training data, we use a synthetically curated dataset containing occluded objects paired with their whole counterparts. Experiments show that our approach outperforms supervised baselines on established benchmarks. Our model can furthermore be used to significantly improve the performance of existing object recognition and 3D reconstruction methods in the presence of occlusions.

Figures (10)

• Figure 1: Amodal Segmentation and Reconstruction via Synthesis. We present pix2gestalt, a method to synthesize whole objects from only partially visible ones, enabling amodal segmentation, recognition, novel-view synthesis, and 3D reconstruction of occluded objects.
• Figure 2: Whole Objects. Pre-trained diffusion models are able to generate all kinds of whole objects. We show samples conditioned on a category from Stable Diffusion. We leverage this synthesis ability for zero-shot amodal reconstruction and segmentation.
• Figure 3: pix2gestalt is an amodal completion model using a latent diffusion architecture. Conditioned on an input occlusion image and a region of interest, the whole (amodal) form is synthesized, thereby allowing other visual tasks to be performed on it too. For conditioning details, see section \ref{['method:diffusion']}.
• Figure 4: Constructing Training Data. To ensure we only occlude whole objects, we use a heuristic that objects closer to the camera than its neighbors are likely whole objects. The green outline around the object shows where the estimated depth is closer to the camera than the background (the red shows when it is not).
• Figure 5: In-the-wild Amodal Completion and Segmentation. We find that pix2gestalt is able to synthesize whole objects in novel situations, including artistic pieces, images taken by an iPhone, and illusions.