Oh-A-DINO: Understanding and Enhancing Attribute-Level Information in Self-Supervised Object-Centric Representations

TL;DR

The paper investigates whether self-supervised and slot-based object-centric representations retain attribute-level information needed for distinguishing objects in multi-object scenes. It reveals a consistent shortfall in non-geometric cues like colour and material, and proposes Oh-A-DINO, a lightweight augmentation that adds object-centric latents learned from PCA-segmented patches via a VAE to pre-trained SSL features without retraining the backbone. Across synthetic datasets CLEVR and CLEVRTex and a real-world dataset Stanford Cars, Oh-A-DINO improves multi-attribute and appearance-based retrieval, especially for colour and material attributes, demonstrating the value of combining global scene understanding with compact object-centric representations. The findings highlight a practical direction for enhancing downstream object-level reasoning by augmenting large pre-trained representations with specialized object-centric latents.

Abstract

Object-centric understanding is fundamental to human vision and required for complex reasoning. Traditional methods define slot-based bottlenecks to learn object properties explicitly, while recent self-supervised vision models like DINO have shown emergent object understanding. We investigate the effectiveness of self-supervised representations from models such as CLIP, DINOv2 and DINOv3, as well as slot-based approaches, for multi-object instance retrieval, where specific objects must be faithfully identified in a scene. This scenario is increasingly relevant as pre-trained representations are deployed in downstream tasks, e.g., retrieval, manipulation, and goal-conditioned policies that demand fine-grained object understanding. Our findings reveal that self-supervised vision models and slot-based representations excel at identifying edge-derived geometry (shape, size) but fail to preserve non-geometric surface-level cues (colour, material, texture), which are critical for disambiguating objects when reasoning about or selecting them in such tasks. We show that learning an auxiliary latent space over segmented patches, where VAE regularisation enforces compact, disentangled object-centric representations, recovers these missing attributes. Augmenting the self-supervised methods with such latents improves retrieval across all attributes, suggesting a promising direction for making self-supervised representations more reliable in downstream tasks that require precise object-level reasoning.

Figures (34)

• Figure 1: DINOv2 representations and slot-based representations struggle at multi-object instance retrieval, due to weak object-specific features (largely colour) in its embeddings. Our method is able to retrieve more relevant images by combining general scene and local representations. DINOv2 representations excel at retrieving images where multiple attributes match such as shape and size, while retrieval degrades when adding colour. Slot-based representations lack specialisation, where retrievals sometimes match fine-grained features such as colour but often failing to retrieve a similar object altogether. Our method performs the best being able to augment the DINOv2 representation to mitigate its shortcomings.
• Figure 2: Object-level features improve prediction for all attributes, while also improving retrieval of multiple simultaneous attributes. The main improvement is achieved on the colour(texture) attribute, which indicates that our method mitigates for the lacking learning signal in the other methods' representations. Slot-based methods perform worse than the pretrained DINOv2 representations when predicting multiple attributes simultaneously.
• Figure 3: Our method leverages the implicit general object understanding of DINOv2 to extract object-level features for multi-object instance retrieval.B: Traditional slot-based object-centric methods learn slot-representations via cross-attention which provide little inductive bias, while having to compress global and object-level features into a small set of latents. A: We propose combining general self-supervised features with learnt object-level features to obtain an improved latent representation. The object-level features are learnt from image patches making training efficient and the latent space expressive.
• Figure 4: Detailed view of extracting object-level features using PCA. From $t$ images we create a segmentation mask which is used to extract the object image patches. We then learn a latent space of the object patches with a VAE.
• Figure 5: Oh-A-DINOv2 improves over SSL and slot-based features for multi-object instance retrieval. Our method outperforms both self-supervised features and slot-based approaches. The table shows that Oh-A-DINOv2 achieves higher precision and lower error rates on CLEVR and CLEVRTex, indicating that combining DINOv2 with object-level VAE features addresses the limitations of pure SSL features and slot-attention methods.