Bootstrapping Top-down Information for Self-modulating Slot Attention

TL;DR

A novel OCL framework incorporating a top-down pathway that bootstraps the semantics of individual objects and modulates the model to prioritize features relevant to these semantics, which achieves state-of-the-art performance across multiple synthetic and real-world object-discovery benchmarks.

Abstract

Object-centric learning (OCL) aims to learn representations of individual objects within visual scenes without manual supervision, facilitating efficient and effective visual reasoning. Traditional OCL methods primarily employ bottom-up approaches that aggregate homogeneous visual features to represent objects. However, in complex visual environments, these methods often fall short due to the heterogeneous nature of visual features within an object. To address this, we propose a novel OCL framework incorporating a top-down pathway. This pathway first bootstraps the semantics of individual objects and then modulates the model to prioritize features relevant to these semantics. By dynamically modulating the model based on its own output, our top-down pathway enhances the representational quality of objects. Our framework achieves state-of-the-art performance across multiple synthetic and real-world object-discovery benchmarks.

Figures (6)

• Figure 1: The overall pipeline of our framework. A top-down pathway is introduced into slot attention to utilize top-down information. The pathway consists of two parts: bootstrapping top-down knowledge and exploiting them. Firstly, semantic information is bootstrapped from slot attention outputs by mapping slots to discrete codes from a learned codebook through vector quantization. Secondly, slot attention is modulated using these codes and its attention maps, transforming it into a self-modulating module. Inner activations are modulated across channels with codes and across space with centered attention maps. Slot attention is then repeated with these modulated activations, yielding more representative slots.
• Figure 2: Visualization of the codebook ${\mathbb{C}}$ on COCO Mscoco. The results show that the codebook learns to capture recurring semantic concepts in the dataset, such as 'pizza' (code 124), 'sign' (code 496), 'clock' (code 235), 'zebra' (code 207), 'motorcycle' (code 341), 'surfer' (code 352), 'dog' (code 359), and 'skier' (code 343).
• Figure 3: Visualization of the input image, predicted object mask, and attention maps of slot attention before and after self-modulation on COCO Mscoco. Lighter the color, higher the attention score.
• Figure 4: Visualization of the codebook ${\mathbb{C}}$ on COCO Mscoco. The results show that the codebook learns to capture broader semantics other than single object categories, such as supercategory (code 468, 328), top-left patch (code 223), and background (code 236, 133, 508).
• Figure A5: Visualization of the predicted object mask on CLEVR6 johnson2017clevr.