From Vicious to Virtuous Cycles: Synergistic Representation Learning for Unsupervised Video Object-Centric Learning

TL;DR

This work tackles the clash between sharp encoder attention and blurrier decoder reconstructions in unsupervised video object-centric learning. It introduces Synergistic Representation Learning (SRL) with two ternary contrastive losses, Deblurring $\mathcal{L}^{\text{CL-dec}}$ and Denoising $\mathcal{L}^{\text{CL-enc}}$, plus a slot-regularization warm-up and a staged training schedule to enable mutual refinement of encoder and decoder representations. Deblurring uses the encoder's sharp priors to sharpen decoder boundaries, while Denoising leverages decoder coherence to denoise encoder features, bridging the representational gap. Across MOVi-C, MOVi-E, YTVIS and transfer to COCO, SRL achieves state-of-the-art FG-ARI and mBO and improves object dynamics prediction with SlotFormer, demonstrating generalization to both video and static images and robustness across backbones.

Abstract

Unsupervised object-centric learning models, particularly slot-based architectures, have shown great promise in decomposing complex scenes. However, their reliance on reconstruction-based training creates a fundamental conflict between the sharp, high-frequency attention maps of the encoder and the spatially consistent but blurry reconstruction maps of the decoder. We identify that this discrepancy gives rise to a vicious cycle: the noisy feature map from the encoder forces the decoder to average over possibilities and produce even blurrier outputs, while the gradient computed from blurry reconstruction maps lacks high-frequency details necessary to supervise encoder features. To break this cycle, we introduce Synergistic Representation Learning (SRL) that establishes a virtuous cycle where the encoder and decoder mutually refine one another. SRL leverages the encoder's sharpness to deblur the semantic boundary within the decoder output, while exploiting the decoder's spatial consistency to denoise the encoder's features. This mutual refinement process is stabilized by a warm-up phase with a slot regularization objective that initially allocates distinct entities per slot. By bridging the representational gap between the encoder and decoder, SRL achieves state-of-the-art results on video object-centric learning benchmarks. Codes are available at https://github.com/hynnsk/SRL.

Figures (17)

• Figure 1: (a) Vicious cycle in video object-centric learning. Noisy inputs from the encoder render the decoder's reconstruction task ill-posed, reinforcing its tendency to produce blurry, low-frequency outputs. In turn, the corrupted gradient from these blurry outputs lacks the high-frequency detail required to refine the encoder's sharp but noisy features. (b) Virtuous cycle of synergistic representation learning. Our framework transforms this conflict into collaboration. We leverage the encoder's sharp attention maps to deblur the decoder output while denoising the encoder features with the decoder's spatially coherent masks.
• Figure 2: Overview of Synergistic Representation learning. The typical pipeline (top) suffers from a conflict between the encoder's sharp but noisy features ($\bar{\bm{v}}$) and the decoder's spatially coherent but blurry features ($\bar{\bm{z}}$). Our framework breaks this cycle by forcing the two modules to synergistically refine one another: (1) Deblurring path: Encoder's sharp attention map is used to refine the blurry decoded features and (2) Denoising path: Decoder's coherent masks provide a robust signal to denoise the encoder's noisy features. Finally, slot regularization during warm-up establishes a solid foundation for this process by ensuring diverse slot specialization.
• Figure 3: Component ablation study.
• Figure 3: Ablation study on coefficients. For (c), we vary $\lambda^{\text{reg}}$ with fixed $\lambda^{\text{CL}}$, and vice versa for (d).
• Figure 4: Ablation study of hierarchical contrastive objective. Pos., S.Pos., and Time indicate whether the positive set $\mathcal{P}$, the semi-positive set $\mathcal{Q}$, and the temporal sampling strategy are used or not.