Multi-Scale Fusion for Object Representation

TL;DR

This work tackles the scale variation challenge in object-centric learning (OCL) by introducing Multi-Scale Fusion (MSF), which leverages an image pyramid and inter-/intra-scale fusion to provide VAE guidance across object sizes. By decomposing VAE representations into scale-invariant and scale-variant components and fusing information across scales via shared and per-scale codebooks, MSF enhances object separability and slot quality for both transformer-based and diffusion-based OCL methods. Empirical results across synthetic and real-world datasets show consistent improvements in OCL metrics (e.g., ARI, ARIfg, mIoU) and better visualization of object delineation, with ablations highlighting the necessity of both fusion types and appropriate scale settings. Overall, MSF offers a scalable, cross-scale enhancement to VAE-guided OCL that improves object representations with manageable computational overhead.

Abstract

Representing images or videos as object-level feature vectors, rather than pixel-level feature maps, facilitates advanced visual tasks. Object-Centric Learning (OCL) primarily achieves this by reconstructing the input under the guidance of Variational Autoencoder (VAE) intermediate representation to drive so-called \textit{slots} to aggregate as much object information as possible. However, existing VAE guidance does not explicitly address that objects can vary in pixel sizes while models typically excel at specific pattern scales. We propose \textit{Multi-Scale Fusion} (MSF) to enhance VAE guidance for OCL training. To ensure objects of all sizes fall within VAE's comfort zone, we adopt the \textit{image pyramid}, which produces intermediate representations at multiple scales; To foster scale-invariance/variance in object super-pixels, we devise \textit{inter}/\textit{intra-scale fusion}, which augments low-quality object super-pixels of one scale with corresponding high-quality super-pixels from another scale. On standard OCL benchmarks, our technique improves mainstream methods, including state-of-the-art diffusion-based ones. The source code is available on https://github.com/Genera1Z/MultiScaleFusion.

Figures (4)

• Figure 1: Multi-scale fusion is necessary in VAE guidance to OCL training. Upper right: simplified architecture of OCL with VAE guidance. Upper left: single-scale VAE encoding/decoding only represent patterns of some certain scale in high quality/confidence. Lower left and right: by resizing the input, patterns of different scales all fall within the VAE encoder/decoder's comfort-zone; low-quality super-pixels in one scale can be augmented by the high-quality in other scales.
• Figure 2: Qualitative results of OCL unsupervised segmentation. The last row is our MSF.
• Figure 3: Qualitative object separability in VAE guidance with MSF. Take the upper left as an example. The circled cube $a$ in column $\bm{X}_n^{\mathrm{si}}$ row scale1 is vague and noisy, while its corresponding representation $b$ in row scale2 is very clear, thus $b$ can augment $a$, yielding clearer $c$ in column $\Dot{\bm{X}}_n^{\mathrm{si}}$ row scale1. As $c$ only captures scale-invariant information, $d$ in column $\bm{X}_n^{\mathrm{sv}}$ row scale1 supplements scale-variant information, which has too many details and is not object-like. By combining $c$ and $d$, we obtain $e$ in column $\bm{X}_n$ row scale1, making this cube much more separable from the background than in column $X$ the corresponding area. Notations are detailed in Sect. \ref{['sect:vae_guidance']} beginning.
• Figure 4: SSLseg vs OCL vs WMs.