SPOC: Spatially-Progressing Object State Change Segmentation in Video

TL;DR

SPOC introduces spatially-progressing OSC segmentation, a novel task to label, within each video frame, which regions of an object are still actionable versus transformed as an irreversible state change unfolds. The approach combines VLM-guided pseudo-labeling with dynamics constraints (causal ordering and ambiguity resolution) to train a per-verb transformer model that scores mask proposals as actionable, transformed, or background, achieving pixel-level segmentation of intra-object state changes. To support research, WhereToChange provides the first large-scale benchmark with fine-grained intra-object state-change annotations across 10 verbs, enormous training data, and challenging evaluation across seen/novel objects and out-of-distribution datasets. Experiments show SPOC outperforms adapted baselines and yields meaningful progress curves for robotics and AR/MR applications, while also exposing the substantial challenges remaining for state-change–sensitive representations in video.

Abstract

Object state changes in video reveal critical cues about human and agent activity. However, existing methods are limited to temporal localization of when the object is in its initial state (e.g., cheese block) versus when it has completed a state change (e.g., grated cheese), offering no insight into where the change is unfolding. We propose to deepen the problem by introducing the spatially-progressing object state change segmentation task. The goal is to segment at the pixel-level those regions of an object that are actionable and those that are transformed. We show that state-of-the-art VLMs and video segmentation methods struggle at this task, underscoring its difficulty and novelty. As an initial baseline, we design a VLM-based pseudo-labeling approach, state-change dynamics constraints, and a novel WhereToChange benchmark built on in-the-wild Internet videos. Experiments on two datasets validate both the challenge of the new task as well as the promise of our model for localizing exactly where and how fast objects are changing in video. We further demonstrate useful implications for tracking activity progress to benefit robotic agents. Overall, our work positions spatial OSC segmentation as a new frontier task for video understanding: one that challenges current SOTA methods and invites the community to build more robust, state-change-sensitive representations. Project page: https://vision.cs.utexas.edu/projects/spoc-spatially-progressing-osc

Figures (15)

• Figure 1: a) An illustration of the spatially-progressing video OSC segmentation problem: With time, the regions within an object undergo a progressive state-change from actionable to transformed. b) A diverse set of spatially-progressing segmentations for different state-change activities. Red is actionable; green is transformed.
• Figure 2: Overview of $\textsc{SPOC}$.a) Pseudo-label generation (Sec. \ref{['subsec:pseudo_label_gen']}): Given a video of a human performing a state-changing activity, we use off-the-shelf object detection liu2023gdino, mask generation kirillov2023sam and tracking models yang2022deaot to extract a set of region mask proposals $M^{0...K}_{t}$ for each frame $I_t$. We then use CLIP clip to apply similarity-score matching of visual region embeddings with textual state-description embeddings to obtain max-similarity pseudo-labels $L^{k}_t$ for each region. b) OSC dynamics constraints (Sec. \ref{['subsec:constraints']}): We refine the pseudo-labels by incorporating several important dynamics constraints that emphasize the temporal progression of state-change transitions while respecting their causal dynamics. c) Model training (Sec. \ref{['subsec:model_training']}): Using our large-scale pseudo-labeled dataset, we train a video model to classify mask proposals into one of three classes: actionable, transformed, or background. Evaluation is done on manually labeled samples.
• Figure 3: Model architecture (Sec. \ref{['subsec:model_training']})
• Figure 4: Qualitative results. a) SPOC clearly distinguishes between actionable and transformed instances of the state-changing object (coated vs uncoated apple), b) with the ability to generalize to novel unseen objects (slicing lime). In contrast, baseline methods tend to be state-change agnostic with decreased ability to disambiguate object states. c) SPOC also shows good generalization to the challenging out-of-distribution VOST dataset. d) Failure cases arise from singular mask proposals spanning the entire object during transitions (single mask for the full lettuce), affecting the model's intra-object segmentation capability.
• Figure 5: Activity progress curves. We show sample frames from a video sequence with progress curves generated by different methods, where vertical lines indicate the time-steps of sampled frames. Ideal curves should decrease monotonically, and saturate upon reaching the end state. In contrast to goal-based representation learning methods such as VIP ma2022vip and LIV ma2023liv, OSC-based curves accurately track task progress, making them valuable for downstream applications like progress monitoring and robot learning.