Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Masked Multi-Query Slot Attention for Unsupervised Object Discovery

Rishav Pramanik, José-Fabian Villa-Vásquez, Marco Pedersoli

TL;DR

This work considers an object-centric approach in which DINO ViT features are reconstructed via a set of queried representations called slots, and proposes a masking scheme on input features that selectively disregards the background regions, inducing the model to focus more on salient objects during the reconstruction phase.

Abstract

Unsupervised object discovery is becoming an essential line of research for tackling recognition problems that require decomposing an image into entities, such as semantic segmentation and object detection. Recently, object-centric methods that leverage self-supervision have gained popularity, due to their simplicity and adaptability to different settings and conditions. However, those methods do not exploit effective techniques already employed in modern self-supervised approaches. In this work, we consider an object-centric approach in which DINO ViT features are reconstructed via a set of queried representations called slots. Based on that, we propose a masking scheme on input features that selectively disregards the background regions, inducing our model to focus more on salient objects during the reconstruction phase. Moreover, we extend the slot attention to a multi-query approach, allowing the model to learn multiple sets of slots, producing more stable masks. During training, these multiple sets of slots are learned independently while, at test time, these sets are merged through Hungarian matching to obtain the final slots. Our experimental results and ablations on the PASCAL-VOC 2012 dataset show the importance of each component and highlight how their combination consistently improves object localization. Our source code is available at: https://github.com/rishavpramanik/maskedmultiqueryslot

Masked Multi-Query Slot Attention for Unsupervised Object Discovery

TL;DR

This work considers an object-centric approach in which DINO ViT features are reconstructed via a set of queried representations called slots, and proposes a masking scheme on input features that selectively disregards the background regions, inducing the model to focus more on salient objects during the reconstruction phase.

Abstract

Unsupervised object discovery is becoming an essential line of research for tackling recognition problems that require decomposing an image into entities, such as semantic segmentation and object detection. Recently, object-centric methods that leverage self-supervision have gained popularity, due to their simplicity and adaptability to different settings and conditions. However, those methods do not exploit effective techniques already employed in modern self-supervised approaches. In this work, we consider an object-centric approach in which DINO ViT features are reconstructed via a set of queried representations called slots. Based on that, we propose a masking scheme on input features that selectively disregards the background regions, inducing our model to focus more on salient objects during the reconstruction phase. Moreover, we extend the slot attention to a multi-query approach, allowing the model to learn multiple sets of slots, producing more stable masks. During training, these multiple sets of slots are learned independently while, at test time, these sets are merged through Hungarian matching to obtain the final slots. Our experimental results and ablations on the PASCAL-VOC 2012 dataset show the importance of each component and highlight how their combination consistently improves object localization. Our source code is available at: https://github.com/rishavpramanik/maskedmultiqueryslot
Paper Structure (26 sections, 6 equations, 5 figures, 6 tables)

This paper contains 26 sections, 6 equations, 5 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Unsupervised Object Discovery
  4. Self-Supervised learning
  5. Masked Autoencoders
  6. Multi-Query Attention
  7. Methodology
  8. Masking
  9. Multi-Query Slot Attention
  10. Combination of Multiple Queries
  11. Training
  12. Inference
  13. Architecture
  14. Encoding
  15. Decoding
  16. ...and 11 more sections

Figures (5)

  • Figure 1: Overview of our proposed architecture. The top part describes the training pipeline that highlights two of our main contributions: Patch Masking Strategy and Multi-Query Attention. The bottom part illustrates the inference phase, which unlike the training phase, does not include the patch masking module and includes an additional Fusion module to perform Slot Alignment via Hungarian Matching to produce the output slots.
  • Figure 2: Masking visualization. These figures show the original image (first row) and the corresponding masked version used during training. As expected, masking removes a large portion of the background leaving only parts of the object of interest and forcing the model to reconstruct the missing parts.
  • Figure 3: Segmentation masks created with different slot attention configurations. From left to right: Original image, normal Slot Attention, Slot Attention with our masking approach, Slot Attention with masking and Multi-Queries (Ours). All configurations involve a ViT-S/16 pre-trained with DINO as the encoder backbone.
  • Figure 4: Segmentation masks of the different heads for the multi-query approach. The first column on the left is the original Image. The next 8 images show the different segmentations for each head and the rightmost column shows the combined results with Hungarian matching.
  • Figure 5: Ablation study concerning the use of different patch tokens namely key, query and value.