Reconstruction-Guided Slot Curriculum: Addressing Object Over-Fragmentation in Video Object-Centric Learning

Abstract

Video Object-Centric Learning seeks to decompose raw videos into a small set of object slots, but existing slot-attention models often suffer from severe over-fragmentation. This is because the model is implicitly encouraged to occupy all slots to minimize the reconstruction objective, thereby representing a single object with multiple redundant slots. We tackle this limitation with a reconstruction-guided slot curriculum (SlotCurri). Training starts with only a few coarse slots and progressively allocates new slots where reconstruction error remains high, thus expanding capacity only where it is needed and preventing fragmentation from the outset. Yet, during slot expansion, meaningful sub-parts can emerge only if coarse-level semantics are already well separated; however, with a small initial slot budget and an MSE objective, semantic boundaries remain blurry. Therefore, we augment MSE with a structure-aware loss that preserves local contrast and edge information to encourage each slot to sharpen its semantic boundaries. Lastly, we propose a cyclic inference that rolls slots forward and then backward through the frame sequence, producing temporally consistent object representations even in the earliest frames. All combined, SlotCurri addresses object over-fragmentation by allocating representational capacity where reconstruction fails, further enhanced by structural cues and cyclic inference. Notable FG-ARI gains of +6.8 on YouTube-VIS and +8.3 on MOVi-C validate the effectiveness of SlotCurri. Our code is available at github.com/wjun0830/SlotCurri.

Figures (11)

• Figure 1: Motivation and method overview. (a) Original video frames. (b) Ground-truth masks. (c) Our baseline SlotContrast slotcontrast learns to decompose with a high slot budget from scratch. This approach often results in over-fragmentation, splitting a single object (car) across multiple slots (brown and purple slots). (d) In contrast, our curriculum learning significantly reduces this over-fragmentation, yielding semantically coherent slots. (e) This is achieved by starting with a minimal slot budget (e.g., 2 slots) and progressively spawning new slots in regions that existing slots fail to capture, thereby preserving object-level slots.
• Figure 2: An architectural overview of our framework.
• Figure 3: Illustration of our reconstruction‑guided slot curriculum learning. The model begins with a few coarse slots that coarsely segment large regions. At each scheduled iteration, slots whose reconstruction errors ($\delta$) remain high are duplicated and perturbed with distance‑aware noise (\ref{['eq.distance-aware-noise']}), creating new slots that focus on the under‑explained areas. Iterating this process across training stages gradually enlarges the slot set, yielding a fine‑grained partition with clearly separated object regions.
• Figure 4: Slot attention maps at an early training (Curriculum stage 1 with 2 slots). (a) Without structural loss, slots broadly capture coarse semantic divisions but exhibit blurred boundaries and partial fragmentation of object parts (helmet of person on the left), indicating weak structural consistency. (b) With structural loss, slots form clearer, sharper boundaries and more coherent object representations, significantly improving object grouping quality.
• Figure 5: Illustration of cyclic inference. Slots on blue background are the output set preserving contextual information.