Are We Done with Object-Centric Learning?

TL;DR

This work argues that modern pixel-space object segmentation models can largely replace slot-based object-centric learning for the core goal of object decomposition. It introduces OCCAM, a training-free probe that leverages object masks to improve robustness against spurious background cues, connecting object-centric representations to downstream OOD generalization. Empirically, HQES and SAM surpass traditional OCL methods on unsupervised object discovery, and OCCAM achieves strong, training-free robustness across multiple spurious-background benchmarks when foreground objects are correctly identified. The authors also provide a practical toolbox and highlight the need for downstream benchmarks and theory to assess the true benefits of object-centric representations in real-world tasks and cognitive study contexts.

Abstract

Object-centric learning (OCL) seeks to learn representations that only encode an object, isolated from other objects or background cues in a scene. This approach underpins various aims, including out-of-distribution (OOD) generalization, sample-efficient composition, and modeling of structured environments. Most research has focused on developing unsupervised mechanisms that separate objects into discrete slots in the representation space, evaluated using unsupervised object discovery. However, with recent sample-efficient segmentation models, we can separate objects in the pixel space and encode them independently. This achieves remarkable zero-shot performance on OOD object discovery benchmarks, is scalable to foundation models, and can handle a variable number of slots out-of-the-box. Hence, the goal of OCL methods to obtain object-centric representations has been largely achieved. Despite this progress, a key question remains: How does the ability to separate objects within a scene contribute to broader OCL objectives, such as OOD generalization? We address this by investigating the OOD generalization challenge caused by spurious background cues through the lens of OCL. We propose a novel, training-free probe called Object-Centric Classification with Applied Masks (OCCAM), demonstrating that segmentation-based encoding of individual objects significantly outperforms slot-based OCL methods. However, challenges in real-world applications remain. We provide the toolbox for the OCL community to use scalable object-centric representations, and focus on practical applications and fundamental questions, such as understanding object perception in human cognition. Our code is available here: https://github.com/AlexanderRubinstein/OCCAM.

Figures (5)

• Figure 1: Where Should We Go? Object-centric learning (OCL) has focused on developing unsupervised mechanisms to separate the representation space into discrete slots. However, the inherent challenges of this task have led to comparatively less emphasis on exploring downstream applications and exploring fundamental benefits. Here, we introduce simple, effective OCL mechanisms by separating objects in pixel space and encoding them independently. We present a case study that demonstrates the downstream advantages of our approach for mitigating spurious correlations. We outline the need to develop benchmarks aligned with fundamental goals of OCL, and explore the downstream efficacy of OCL representations.
• Figure 2: Overview of Object-Centric Classification with Applied Masks (OCCAM). There are two main parts. The first part (§ \ref{['sec:method:generate_representations']}) uses entity segmentation masks for object-centric representation generation. The second part (§ \ref{['sec:method:robust_classifier']}) performs robust classification by selecting representations corresponding to the foreground object and using them for classification. Indices $[i_0, \ldots, i_k, \ldots]$ correspond to each object in the scene.
• Figure 3: Qualitative Results on Object Discovery. Dinosaur, SlotDiffusion, and FT-Dinosaur are existing object-centric learning (OCL) approaches. Sam and HQES refer to zero-shot segmentation methods. Images are from Movi-E. Sam and HQES masks fit objects much better than the masks predicted by OCL methods. All columns except for HQES are taken from ft-dinosaur.
• Figure 4: Foreground Object Detection. ROC-curves for foreground detection methods. For each scoring scheme, we measure how well the true foreground objects in the ImageNet-validation dataset are detected. More details in § \ref{['subsec:ood_detection']}.
• Figure 5: Gaps in accuracies [Common - Counter] for Common and Counter subsets of CounterAnimals CounterAnimals dataset correspondingly for different CLIP models and pre-training datasets. "Gap" results are computed for CLIP CLIP zero-shot performance without using any masks; "Gap-FG" results are computed when using OCCAM with HQES EntitySeg masks, Class-Aided foreground selection method, and "Gray BG + Crop" mask applying operation.