Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Attention Normalization Impacts Cardinality Generalization in Slot Attention

Markus Krimmel, Jan Achterhold, Joerg Stueckler

TL;DR

It is demonstrated that design decisions on normalizing the aggregated values in the attention architecture have considerable impact on the capabilities of Slot Attention to generalize to a higher number of slots and objects as seen during training.

Abstract

Object-centric scene decompositions are important representations for downstream tasks in fields such as computer vision and robotics. The recently proposed Slot Attention module, already leveraged by several derivative works for image segmentation and object tracking in videos, is a deep learning component which performs unsupervised object-centric scene decomposition on input images. It is based on an attention architecture, in which latent slot vectors, which hold compressed information on objects, attend to localized perceptual features from the input image. In this paper, we demonstrate that design decisions on normalizing the aggregated values in the attention architecture have considerable impact on the capabilities of Slot Attention to generalize to a higher number of slots and objects as seen during training. We propose and investigate alternatives to the original normalization scheme which increase the generalization capabilities of Slot Attention to varying slot and object counts, resulting in performance gains on the task of unsupervised image segmentation. The newly proposed normalizations represent minimal and easy to implement modifications of the usual Slot Attention module, changing the value aggregation mechanism from a weighted mean operation to a scaled weighted sum operation.

Attention Normalization Impacts Cardinality Generalization in Slot Attention

TL;DR

It is demonstrated that design decisions on normalizing the aggregated values in the attention architecture have considerable impact on the capabilities of Slot Attention to generalize to a higher number of slots and objects as seen during training.

Abstract

Object-centric scene decompositions are important representations for downstream tasks in fields such as computer vision and robotics. The recently proposed Slot Attention module, already leveraged by several derivative works for image segmentation and object tracking in videos, is a deep learning component which performs unsupervised object-centric scene decomposition on input images. It is based on an attention architecture, in which latent slot vectors, which hold compressed information on objects, attend to localized perceptual features from the input image. In this paper, we demonstrate that design decisions on normalizing the aggregated values in the attention architecture have considerable impact on the capabilities of Slot Attention to generalize to a higher number of slots and objects as seen during training. We propose and investigate alternatives to the original normalization scheme which increase the generalization capabilities of Slot Attention to varying slot and object counts, resulting in performance gains on the task of unsupervised image segmentation. The newly proposed normalizations represent minimal and easy to implement modifications of the usual Slot Attention module, changing the value aggregation mechanism from a weighted mean operation to a scaled weighted sum operation.
Paper Structure (54 sections, 5 theorems, 33 equations, 15 figures, 7 tables, 2 algorithms)

This paper contains 54 sections, 5 theorems, 33 equations, 15 figures, 7 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Slot Attention
  4. Von Mises-Fisher Distributions
  5. Slot Attention and von Mises-Fisher Mixture Model Parameter Estimation
  6. Relating Slot Attention to EM
  7. E-Step
  8. M-Step
  9. Comparing Normalizations in EM Analogy
  10. Weighted Mean
  11. Layer Normalization
  12. Weighted Sum
  13. Methods of Normalization
  14. Weighted Sum Normalization with Fixed Scaling
  15. Weighted Sum Normalization with Batch Scaling
  16. ...and 39 more sections

Key Result

Proposition 1

Consider Slot Attention with weighted mean normalization and any fixed model parameters and fixed input data $\tilde{\bm{x}}_1,...,\tilde{\bm{x}}_N$ with $N \geq 1$. Then, there exists no function $f: \mathbb{R}^D \to \mathbb{R}$ such that it holds for arbitrary $K \geq 1$, arbitrary slots $\tilde{\bm{\theta}}_1,...,\tilde{\bm{\theta}}_K$ and resulting normalized update codes $\bm{u}_1,...,\bm{u}

Figures (15)

  • Figure 1: Dependence of performance on slot and object count. Models are trained on CLEVR6 with 7 slots. Note the non-zero y-intercept.
  • Figure 2: Qualitative results on a CLEVR10 images, showing reconstructions and (soft) segmentations. The models are trained on CLEVR6 with 7 slots and evaluated with 21 slots.
  • Figure 3: Dependence of segmentation performance on slot and object count. Models are trained on CLEVR6 with 11 slots.
  • Figure 4: Dependence of segmentation performance on slot and object count. Models are trained on MOVi-C10 with 11 slots.
  • Figure 5: Dependence of segmentation performance on slot and object count. Models are trained on MOVi-C6 with 7 slots.
  • ...and 10 more figures

Theorems & Definitions (10)

  • Proposition 1
  • Proposition 2
  • Lemma 1
  • proof
  • Lemma 2
  • proof
  • Lemma 3
  • proof
  • proof
  • proof