SynergyAmodal: Deocclude Anything with Text Control

TL;DR

SynergyAmodal tackles the data scarcity bottleneck in open-world amodal completion by fusing in-the-wild modal data, human plausibility guidance, and strong generative priors into a three-stage pipeline. It first learns a self-supervised partial completion model from diverse images, then uses human-in-the-loop co-synthesis and priors to produce 16K high-quality amodal pairs, and finally trains a text-conditioned full completion diffusion model (DeoccAnything) on the synthesized data. The approach yields state-of-the-art zero-shot generalization and explicit text controllability for amodal completion, validated by quantitative metrics on COCOA/BSDSA and qualitative analyses, while enabling downstream 3D reconstruction potential. The dataset SynergyAmodal16K and the DeoccAnything model are released to support broader research in amodal perception and AIGC applications.

Abstract

Image deocclusion (or amodal completion) aims to recover the invisible regions (\ie, shape and appearance) of occluded instances in images. Despite recent advances, the scarcity of high-quality data that balances diversity, plausibility, and fidelity remains a major obstacle. To address this challenge, we identify three critical elements: leveraging in-the-wild image data for diversity, incorporating human expertise for plausibility, and utilizing generative priors for fidelity. We propose SynergyAmodal, a novel framework for co-synthesizing in-the-wild amodal datasets with comprehensive shape and appearance annotations, which integrates these elements through a tripartite data-human-model collaboration. First, we design an occlusion-grounded self-supervised learning algorithm to harness the diversity of in-the-wild image data, fine-tuning an inpainting diffusion model into a partial completion diffusion model. Second, we establish a co-synthesis pipeline to iteratively filter, refine, select, and annotate the initial deocclusion results of the partial completion diffusion model, ensuring plausibility and fidelity through human expert guidance and prior model constraints. This pipeline generates a high-quality paired amodal dataset with extensive category and scale diversity, comprising approximately 16K pairs. Finally, we train a full completion diffusion model on the synthesized dataset, incorporating text prompts as conditioning signals. Extensive experiments demonstrate the effectiveness of our framework in achieving zero-shot generalization and textual controllability. Our code, dataset, and models will be made publicly available at https://github.com/imlixinyang/SynergyAmodal.

Figures (19)

• Figure 1: Point-prompted amodal completion by combining SAM kirillov2023segment and the proposed DeoccAnything model.
• Figure 2: Framework. We first utilize modal data (i.e., EntitySeg qi2022high) to train a partial completion model using the proposed self-supervised deocclusion learning algorithm. Next, we employ a data-human-model collaborative pipeline to generate high-quality amodal data with rich annotations. Finally, we train a full completion model using the synthesized amodal data.
• Figure 3: Global-to-local inference strategy.
• Figure 4: Effect of different variation number.
• Figure 5: Effect of different occlusion percentages.