Vector-Quantized Vision Foundation Models for Object-Centric Learning

TL;DR

The paper tackles object-centric learning (OCL) by addressing texture-heavy object segmentation through a unified approach: Vector-Quantized Vision Foundation Models for OCL (VVO). It introduces shared quantized reconstruction targets derived from the same Vision Foundation Model features to supervise aggregation, enabling direct use of VFM features for slot formation and benefiting a range of decoders (mixture, auto-regressive, diffusion). Across synthetic and real datasets, VVO consistently improves object-discovery metrics and downstream prediction/reasoning on Physion, with ablations showing the advantages of shared VFM encoding and the proposed Q-quantization tricks. The work provides both theoretical insight and practical gains, and releases the code and checkpoints to serve as a general testbed for integrating VFMs with OCL.

Abstract

Object-Centric Learning (OCL) aggregates image or video feature maps into object-level feature vectors, termed \textit{slots}. It's self-supervision of reconstructing the input from slots struggles with complex object textures, thus Vision Foundation Model (VFM) representations are used as the aggregation input and reconstruction target. Existing methods leverage VFM representations in diverse ways yet fail to fully exploit their potential. In response, we propose a unified architecture, Vector-Quantized VFMs for OCL (VQ-VFM-OCL, or VVO). The key to our unification is simply shared quantizing VFM representations in OCL aggregation and decoding. Experiments show that across different VFMs, aggregators and decoders, our VVO consistently outperforms baselines in object discovery and recognition, as well as downstream visual prediction and reasoning. We also mathematically analyze why VFM representations facilitate OCL aggregation and why their shared quantization as reconstruction targets strengthens OCL supervision. Our source code and model checkpoints are available on https://github.com/Genera1Z/VQ-VFM-OCL.

Figures (7)

• Figure 1: We utilize two observations: (i) Compared to non-VFMs, VFMs extract features with better object separability, i.e., smaller intra-object distances and larger inter-object distances $\Rightarrow$ We facilitate OCL aggregation via VFM features; (ii) VFM and non-VFM features have a distribution gap, i.e., separated centroids $\Rightarrow$ We strengthen OCL supervision by reconstructing the quantized features shared from the same VFM rather than from another encoder.
• Figure 2: VVO is a unified architecture that fully utilizes VFMs in OCL. It not only extracts VFM features with better objectness to facilitate object information aggregation; but further quantizes those VFM features as reconstruction targets to strengthen OCL training supervision. With typical SlotAttention or it variants as the aggregator and Vector-Quantization as the quantizer, VVO supports different VFMs as the encoder, and supports mainstream mixture, auto-regression and diffusion models as the decoder.
• Figure 3: Model architecture comparison.
• Figure 4: Qualitative object discovery performance comparison.
• Figure 5: Visual prediction (upper) and reasoning (lower) performance on Physion (#slot=8). VVO has smaller prediction error in all time steps, and higher reasoning accuracy in later time steps.