Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Interpreting the structure of multi-object representations in vision encoders

Tarun Khajuria, Braian Olmiro Dias, Marharyta Domnich, Jaan Aru

TL;DR

This work addresses how vision encoders represent multiple objects by formalizing two structural properties: object-binding into discrete tokens ($M_1 = A_{ss}$) and object-wise token segregation ($M_2 = \frac{A_{sp}}{A_{pp}}$). It introduces paired-object and global-probe experiments on COCO across diverse encoders (ViT, CLIP/BLIP/FLAVA, and DINO variants) to quantify these properties via object decoding from object-specific tokens versus the CLS token. Key findings show object-specific tokens capture discriminative object information and retain some object-specific detail even for less emphasized background objects, while CLS tends to encode scene-level information but may entangle multiple objects and underperform for background objects; the training objective and architecture strongly shape these representations. The paper contributes practical measures to guide encoder selection and layer-wise decoder/adaptor design for downstream tasks requiring fine-grained object information, enabling better handling of multi-object scenes in real-world applications.

Abstract

In this work, we interpret the representations of multi-object scenes in vision encoders through the lens of structured representations. Structured representations allow modeling of individual objects distinctly and their flexible use based on the task context for both scene-level and object-specific tasks. These capabilities play a central role in human reasoning and generalization, allowing us to abstract away irrelevant details and focus on relevant information in a compact and usable form. We define structured representations as those that adhere to two specific properties: binding specific object information into discrete representation units and segregating object representations into separate sets of tokens to minimize cross-object entanglement. Based on these properties, we evaluated and compared image encoders pre-trained on classification (ViT), large vision-language models (CLIP, BLIP, FLAVA), and self-supervised methods (DINO, DINOv2). We examine the token representations by creating object-decoding tasks that measure the ability of specific tokens to capture individual objects in multi-object scenes from the COCO dataset. This analysis provides insights into how object-wise representations are distributed across tokens and layers within these vision encoders. Our findings highlight significant differences in the representation of objects depending on their relevance to the pre-training objective, with this effect particularly pronounced in the CLS token (often used for downstream tasks). Meanwhile, networks and layers that exhibit more structured representations retain better information about individual objects. To guide practical applications, we propose formal measures to quantify the two properties of structured representations, aiding in selecting and adapting vision encoders for downstream tasks.

Interpreting the structure of multi-object representations in vision encoders

TL;DR

This work addresses how vision encoders represent multiple objects by formalizing two structural properties: object-binding into discrete tokens () and object-wise token segregation (). It introduces paired-object and global-probe experiments on COCO across diverse encoders (ViT, CLIP/BLIP/FLAVA, and DINO variants) to quantify these properties via object decoding from object-specific tokens versus the CLS token. Key findings show object-specific tokens capture discriminative object information and retain some object-specific detail even for less emphasized background objects, while CLS tends to encode scene-level information but may entangle multiple objects and underperform for background objects; the training objective and architecture strongly shape these representations. The paper contributes practical measures to guide encoder selection and layer-wise decoder/adaptor design for downstream tasks requiring fine-grained object information, enabling better handling of multi-object scenes in real-world applications.

Abstract

In this work, we interpret the representations of multi-object scenes in vision encoders through the lens of structured representations. Structured representations allow modeling of individual objects distinctly and their flexible use based on the task context for both scene-level and object-specific tasks. These capabilities play a central role in human reasoning and generalization, allowing us to abstract away irrelevant details and focus on relevant information in a compact and usable form. We define structured representations as those that adhere to two specific properties: binding specific object information into discrete representation units and segregating object representations into separate sets of tokens to minimize cross-object entanglement. Based on these properties, we evaluated and compared image encoders pre-trained on classification (ViT), large vision-language models (CLIP, BLIP, FLAVA), and self-supervised methods (DINO, DINOv2). We examine the token representations by creating object-decoding tasks that measure the ability of specific tokens to capture individual objects in multi-object scenes from the COCO dataset. This analysis provides insights into how object-wise representations are distributed across tokens and layers within these vision encoders. Our findings highlight significant differences in the representation of objects depending on their relevance to the pre-training objective, with this effect particularly pronounced in the CLS token (often used for downstream tasks). Meanwhile, networks and layers that exhibit more structured representations retain better information about individual objects. To guide practical applications, we propose formal measures to quantify the two properties of structured representations, aiding in selecting and adapting vision encoders for downstream tasks.
Paper Structure (23 sections, 2 equations, 5 figures, 2 tables)

This paper contains 23 sections, 2 equations, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Evaluating a trained model's capabilities
  4. Interpreting model's representations
  5. Methods
  6. Formal Definition
  7. Experimental Setup: Paired Object Task
  8. Experimental Setup: Global Probe
  9. Dataset
  10. Linear Probing
  11. Analysed models and their configuration
  12. Results
  13. A snapshot of network's global and image-specific representation
  14. How different token representations encode objects, their interaction, and their importance
  15. Variation of representations across models with different training objectives and architectures: key insights
  16. ...and 8 more sections

Figures (5)

  • Figure 1: A. Explanation of how the token representations are obtained. We analyse four kinds of tokens in this study: 1) CLS token: The special token usually used in models for downstream tasks; 2) Avg_obj(Object-specific token): obtained by averaging the token representations of the object-masked tokens as shown in the figure. 3) Random_obj (Object-specific token): Rather than averaging, we sample one of the tokens from the masked token space of the object 4) Random: Obtained by sampling any random token from the token space other than the CLS token B. Describes the experimental setup in which we perform decoding in paired object tasks; each object-specific representation decodes 1) the object itself, 2) the other object in the image, and 3) the combination of both objects. C. Shows a sample paired object decoding task; given an image, the task is to decode if it contains object1 (cat/dog), object2 (chair/couch) or a combination of both.
  • Figure 2: a. Paired object decoding task results for BLIP across layers: Average decoding performance for different layers (y-axis) and token types (x-axis) over 6 tasks for BLIP. In the subfigures, the y-axis contains variations of where the object-specific tokens (random_obj and avg_obj) are obtained. The different columns show results for 1) decoding the primary object 2) the secondary object 3) the combination of both objects in the image. The decoding pattern remains after averaging, with the tokens from the objects modelling the most useful information for categorising the objects. The object-specific tokens are much better than the CLS token, which has to capture the larger scene context. b. Visualisation of cosine similarity of highlighted token to other tokens for a token from primary and secondary objects at various layers of BLIP model.
  • Figure 3: Layer-wise test set decoding accuracy for primary and secondary objects for pre-trained models in the study. Results for all models are shown in the appendix. The accuracies are averaged over the six object sets. In each sub-graph, the y-axis denotes the decoding accuracy, and the x-axis denotes the layer at which the accuracy was observed. We observe consistent decoding trends across models with a few variations reported in Section \ref{['sec:Variation across models']}.
  • Figure 4: a. Variation in decoding accuracy between instances of objects ‘in caption’ and ‘not in caption’. Each subplot represents the decoding of the object by its object-specific representation. b. Object detection accuracy on the secondary objects in paired using probes trained on three different token representations from CLIP and zero-shot CLIP accuracy.
  • Figure 5: The figure shows both measures of modularity evaluated for all vision encoders. Along with this, the figure has the accuracy of global probing task for avg_obj token evaluated for 'In caption': Objects mentioned in the caption and 'Not in caption': Objects not mentioned in the caption. The results for each network are from the layer with the best $M_1$ across the layers. We see how the first measure of modularity $M_1$ correlates highly with the accuracy of decoding background tokens across networks (Pearson's correlation coefficient = 0.94, p = 0.001). This also shows that the measures from the paired object task are highly predictive of general representation of objects in the network. The second measure of modularity (Measure of entanglement) $M_2$ correlates (Pearson's coefficient = 0.74, p = 0.02) with the drop in accuracy for objects 'not in the caption' compared to the objects mentioned 'in the caption'. Hence, more modular representations favour better representation of individual background objects.