Generalizing Single-View 3D Shape Retrieval to Occlusions and Unseen Objects

TL;DR

This work addresses single-view 3D shape retrieval under realistic conditions featuring occlusions and unseen shapes/objects. It introduces MOOS, a scalable synthetic occlusion dataset, and a standardized evaluation protocol with both view-independent and view-dependent metrics, alongside a real-data benchmark (Pix3D, Scan2CAD). Using CMIC as a baseline, it shows that pretraining on MOOS and finetuning on real data significantly improves robustness to occlusions and unseen data, enabling better cross-domain transfer and few-shot adaptation. The findings highlight the importance of occlusion-aware pretraining for practical retrieval systems and establish a framework for more realistic benchmarking in 3D shape retrieval.

Abstract

Single-view 3D shape retrieval is a challenging task that is increasingly important with the growth of available 3D data. Prior work that has studied this task has not focused on evaluating how realistic occlusions impact performance, and how shape retrieval methods generalize to scenarios where either the target 3D shape database contains unseen shapes, or the input image contains unseen objects. In this paper, we systematically evaluate single-view 3D shape retrieval along three different axes: the presence of object occlusions and truncations, generalization to unseen 3D shape data, and generalization to unseen objects in the input images. We standardize two existing datasets of real images and propose a dataset generation pipeline to produce a synthetic dataset of scenes with multiple objects exhibiting realistic occlusions. Our experiments show that training on occlusion-free data as was commonly done in prior work leads to significant performance degradation for inputs with occlusion. We find that that by first pretraining on our synthetic dataset with occlusions and then finetuning on real data, we can significantly outperform models from prior work and demonstrate robustness to both unseen 3D shapes and unseen objects.

Figures (17)

• Figure 1: We focus on the single-view 3D shape retrieval task in complex but realistic scenarios where both the object and 3D shape candidates may be unseen during training, and the object may be observed in cluttered scenes under significant occlusions.
• Figure 2: Overview of our single-view 3D shape retrieval training pipeline. We pretrain with a large synthetic dataset exhibiting object occlusions. After fine-tuning on real image datasets we significantly improve generalization to occlusions and previously unseen objects.
• Figure 3: The pipeline of our Multi-Object Occlusion Scenes (MOOS) dataset generation procedure. We sample and iteratively place objects while avoiding collisions. The resulting scenes contain a variety of object categories and exhibit realistic occlusions.
• Figure 4: Plots of evaluation metrics vs. occlusion rates on MOOS seen (blue) and unseen (red) occluded objects. Acc and CD decreases for higher occlusion rates while the LPIPS remains stable.
• Figure 5: Results on MOOS seen (rows 1-2) and unseen (rows 3-5) objects. The target has an orange mask in the 2nd column . For seen objects, we retrieve the matching object despite occlusions. For unseen objects, we see failures for objects with thin structures (row 3 and 5).