Slot-Guided Adaptation of Pre-trained Diffusion Models for Object-Centric Learning and Compositional Generation

TL;DR

This work introduces SlotAdapt, a method that teams slot attention with adapters in a pretrained diffusion model to achieve object-centric learning and compositional image generation without relying on text-conditioned prompts. By injecting adapter cross-attention conditioned on learned slots and adding a register token to capture global context, SlotAdapt preserves the generative power of pretrained diffusion models while reducing text-conditioning bias. A self-supervised guidance loss aligns cross-attention maps from adapters with slot attention, improving object alignment and addressing part-whole ambiguities. Across MOVi-E, VOC, and COCO, SlotAdapt with guidance achieves superior object discovery, segmentation, and compositional editing, highlighting its effectiveness on complex real-world scenes and its potential to advance unsupervised visual reasoning and controllable generation.

Abstract

We present SlotAdapt, an object-centric learning method that combines slot attention with pretrained diffusion models by introducing adapters for slot-based conditioning. Our method preserves the generative power of pretrained diffusion models, while avoiding their text-centric conditioning bias. We also incorporate an additional guidance loss into our architecture to align cross-attention from adapter layers with slot attention. This enhances the alignment of our model with the objects in the input image without using external supervision. Experimental results show that our method outperforms state-of-the-art techniques in object discovery and image generation tasks across multiple datasets, including those with real images. Furthermore, we demonstrate through experiments that our method performs remarkably well on complex real-world images for compositional generation, in contrast to other slot-based generative methods in the literature. The project page can be found at https://kaanakan.github.io/SlotAdapt/.

Figures (18)

• Figure 1: SlotAdapt Architecture We extract object-centric information from the input image using a visual backbone, which combines DINO and slot attention. Stable Diffusion VAE is used to encode the image into latent space and then noise is added to the latent. Diffusion process is conditioned on the generated slots as well as the register token which is generated by (mean) pooling the slots. We use the original cross attention layers of diffusion model to condition on the register token, and additional adapter attentions to condition on the learned slots. The overall objective is to predict the noise added to the image. Additionally, we introduce a guidance loss between the slot attention masks and adapter cross attention masks, which encourages the similarity between them. The guidance is only applied in the third upsampling block, while slot conditioning is applied throughout all downsampling and upsampling blocks.
• Figure 2: Qualitative comparison: with vs. without guidance. We visualize generated images and predicted segments on COCO dataset.
• Figure 3: Unsupervised Object Segmentation.We show visualizations of segments on COCO (left) and VOC (right). SlotAdapt accurately binds distinct instances belonging to the same class.
• Figure 4: Qualitative comparisons with other methods on COCO. We visualize predicted segments of SlotAdapt in comparison to LSD and SlotDiffusion. SlotAdapt can more effectively differentiate between object instances of the same class compared to other methods.
• Figure 5: Generation Results.We show sample images reconstructed by SlotAdapt on COCO (left) and VOC (right). SlotAdapt generates reconstructions highly faithful to the original input images.